Display device and media asset playing method

Abstract

The invention provides a display device and a media asset playing method, and a processor responds to a music recognition instruction input by a user, and obtains a first audio clip of sound data played by a current video. And the server searches audio data of the background music according to the first audio clip. And if the audio content of the audio data exceeds the audio content range of the sound data, controlling the video player to stop playing the current video, and controlling the audio player to play the audio data. And in response to the audio playing stopping instruction, controlling the audio player to stop playing the audio data, and controlling the video player to continuously play the current video on the basis of the first playing progress. In this way, if the user is interested in the background music and willingness to play other segments of the background music, the user can achieve the purpose of audio-video linkage only by inputting the music recognition instruction, and therefore the use experience of the user is improved.

Description

Display device and media playback method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of application number 202211634779.3 filed with the China Patent Office on December 19, 2022, application number 202211630520.1 filed with the China Patent Office on December 19, 2022, application number 202211649279.7 filed with the China Patent Office on December 20, 2022, and application number 202211666085.8 filed with the China Patent Office on December 23, 2022, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of communication technology, and in particular to a display device and a media playback method. Background Art

[0004] Display devices can use a local player or a player provided by an internet browser to play videos such as television programs and movies. For example, after a user opens a video link in a video application, the video corresponding to the link is played using the video application's built-in player. While the user is watching the video, the video may be configured with background music.

[0005] Background music for a video typically only plays a portion of the music. If a user becomes interested in the music and wishes to play a different section, they must manually pause the video, retrieve basic information about the music, and then use that information to search for the rest of the music on the music app. After listening to the rest of the music, the user returns to the video playback page and manually clicks to continue playing. This complex method of media playback results in a poor user experience.

[0006] Summary of the Invention

[0007] In a first aspect, a display device according to an embodiment of the present application includes: a display for displaying an image and / or a user interface; an audio player for playing audio data; a user input interface for receiving an input signal; a memory configured to store computer instructions and data associated with the display, at least one processor connected to the display, the audio player, the user input interface and the memory, and configured to execute the computer instructions so that the display device executes: in response to a received song identification instruction, collecting a first audio segment of the sound data of the current video playback, and sending a song identification request containing the first audio segment to a server, so that the server searches for the sound data according to the first audio segment The method comprises the steps of: receiving audio data from the server, wherein the first audio segment represents a segment of the sound data, and the audio data represents audio content that is similar to or identical to the sound data; if the audio content of the audio data exceeds the audio content range of the sound data, receiving the audio data fed back by the server, controlling the video player to pause playing the current video, recording the first playback progress of the current video when the current video is paused, and controlling the audio player to play the audio data; in response to the received audio stop playback instruction, controlling the audio player to stop playing the audio data, and controlling the video player to continue playing the current video based on the first playback progress.

[0008] In the second aspect, a media resource playback method according to an embodiment of the present application includes: in response to a received song identification instruction, collecting the first audio segment of the sound data of the current video playback, and sending a song identification request containing the first audio segment to the server, so that the server searches for the audio data of the sound data based on the first audio segment, wherein the first audio segment represents a segment of the sound data, and the audio data represents audio content similar to or identical to the sound data; if the audio content of the audio data exceeds the audio content range of the sound data, receiving the audio data feedback from the server, controlling the video player to pause playing the current video, recording the first playback progress of the current video when the current video is paused, and controlling the audio player to play the audio data; in response to a received audio stop playback instruction, controlling the audio player to stop playing the audio data, and controlling the video player to continue playing the current video based on the first playback progress. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG1 illustrates an operation scenario between a display device and a control apparatus according to some embodiments;

[0010] FIG2 is a block diagram of a hardware configuration of a control device 100 according to some embodiments;

[0011] FIG3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments;

[0012] FIG4 is a diagram illustrating a software configuration in the display device 200 according to some embodiments;

[0013] FIG5 is a schematic diagram of a user interface of a display device according to some embodiments;

[0014] FIG6 is a schematic diagram of a user interface of another display device according to some embodiments;

[0015] FIG7 is a schematic diagram of a user interface of another display device according to some embodiments;

[0016] FIG8 is a schematic diagram of a media asset playback system architecture according to some embodiments;

[0017] FIG9 is a schematic diagram of a user interface of another display device according to some embodiments;

[0018] FIG10 is a schematic diagram illustrating a principle of the relationship between sound data and audio data according to some embodiments;

[0019] FIG11 is a schematic diagram showing a principle diagram of the relationship between sound data and audio data according to some embodiments;

[0020] FIG12 is a schematic diagram of a user interface of another display device according to some embodiments;

[0021] FIG13 is a schematic diagram showing a principle diagram of the relationship between sound data and audio data according to some embodiments;

[0022] FIG14 is a schematic diagram showing a principle diagram of the relationship between sound data and audio data according to some embodiments;

[0023] FIG15 is a signaling diagram of a media asset playback method according to some embodiments;

[0024] FIG16 is a signaling diagram of another method for playing media resources according to some embodiments;

[0025] FIG17 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0026] FIG18 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0027] FIG19 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0028] FIG20 is a flow chart of a method for playing media resources according to some embodiments;

[0029] FIG21 is a timing diagram illustrating audio recognition in a display device according to some embodiments;

[0030] FIG22 is a schematic diagram illustrating a user's request to a display device to recognize audio according to some embodiments;

[0031] FIG23 is a schematic diagram of an application scenario in which a display device is located according to some embodiments;

[0032] FIG24 is a timing diagram showing audio recognition by a display device according to some embodiments;

[0033] FIG25 is a flowchart illustrating determination of first audio data and third audio data according to some embodiments;

[0034] FIG26 is a timing diagram showing audio recognition by a display device according to some embodiments;

[0035] FIG27 is a timing diagram illustrating an audio recognition operation performed by a display device through a recognition audio server according to some embodiments;

[0036] FIG28 is a schematic diagram of a song information recognition system architecture according to some embodiments;

[0037] FIG29 is a schematic diagram of a user interface of a display device according to some embodiments;

[0038] FIG30 is a signaling diagram of a song information identification method process according to some embodiments;

[0039] FIG31 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0040] FIG32 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0041] FIG33 is a schematic diagram illustrating the principle of filtering songs based on a TV series configuration file according to some embodiments;

[0042] FIG34 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0043] FIG35 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0044] FIG36 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0045] FIG37 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0046] FIG38 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0047] FIG39 is a signaling diagram of a specific implementation process of song information identification according to some embodiments;

[0048] FIG40 is a flow chart of a song information identification method according to some embodiments;

[0049] FIG41 is a flow chart of another song information identification method according to some embodiments;

[0050] FIG42 is a schematic diagram showing an icon control interface of an application in a display device according to some embodiments;

[0051] FIG43 is a system framework diagram for performing music recommendation according to some embodiments;

[0052] FIG44 is a flowchart illustrating a music recommendation method applied to a display device according to some embodiments;

[0053] FIG45 is a schematic diagram of an interaction process between a display device and a server according to some embodiments;

[0054] FIG46 is a schematic diagram of a first display interface when a display shows a target recommended song list according to some embodiments;

[0055] FIG47 is a schematic diagram of a second display interface when a display shows a target recommended song list according to some embodiments;

[0056] FIG48 is a schematic diagram of an interaction process between a display device and a server in a film and television playback scenario according to some embodiments;

[0057] FIG49 is a schematic diagram of an interaction process between a display device and a server in another film and television playback scenario according to some embodiments;

[0058] FIG50 is a schematic diagram of an interaction process between a display device and a server in a song playback scenario according to some embodiments;

[0059] Figure 51 is a flow chart of a music recommendation method applied to a server according to some embodiments. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of some embodiments of the present application more clear, the following will clearly and completely describe the technical solutions of some embodiments of the present application in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all of the embodiments.

[0061] It should be noted that the brief descriptions of terms in some embodiments of the present application are only for the purpose of facilitating understanding of the embodiments described below, and are not intended to limit the embodiments of some embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0062] In some embodiments of the present application, the terms "first," "second," "third," and the like in the description and claims and the accompanying drawings are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or precedence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable where appropriate.

[0063] The term "remote control" as used in this application refers to a component of a display device (such as the display device disclosed in this application) that can typically wirelessly control the display device over a short distance. It typically connects to the display device via infrared and / or radio frequency (RF) signals and / or Bluetooth, and may also include functional modules such as WiFi, wireless USB, Bluetooth, and motion sensors. For example, a handheld touch remote control replaces most of the physical built-in hard keys in a typical remote control device with a user interface on a touch screen.

[0064] The electronic device in this application can be a display device or other electronic device with a voice assistant function. The following uses a display device as an example to illustrate the solution. Figure 1 is a schematic diagram of an operation scenario between a display device and a terminal device provided in some embodiments of this application. As shown in Figure 1, a user can operate the display device 200 through a mobile terminal 300 and a terminal device 100.

[0065] In some embodiments, the terminal device 100 may be a remote control, and the communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, etc., to control the display device 200 wirelessly or by other wired means. The user can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0066] In some embodiments, the mobile terminal 300 can install software applications on the display device 200 and connect and communicate via a network communication protocol to achieve one-to-one control operations and data communication. Audio and video content displayed on the mobile terminal 300 can also be transmitted to the electronic device 200 for synchronized display.

[0067] 1 , the display device 200 also communicates data with the server 400 via various communication methods. The display device 200 may be connected to the server 400 via a local area network (LAN), a wireless local area network (WLAN), or other networks.

[0068] In addition to providing broadcast reception television functions, the display device 200 may also provide intelligent network television functions with computer support functions, including but not limited to network television, smart TV, Internet Protocol television (IPTV), etc.

[0069] FIG2 is a hardware configuration block diagram of the display device 200 in FIG1 provided in some embodiments of the present application.

[0070] In some embodiments, the display apparatus 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, at least one processor 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0071] In some embodiments, the detector 230 is used to collect signals from the external environment or from interactions with the external environment.

[0072] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0073] In some embodiments, the communicator 220 is a component for communicating with an external device or the server 400 according to various communication protocol types.

[0074] In some embodiments, the processor 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The processor 250 controls the overall operation of the display device 200.

[0075] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0076] In some embodiments, the user input interface 280 is an interface that can be used to receive control input.

[0077] Figure 3 is a block diagram of the hardware configuration of the terminal device in Figure 1 provided in some embodiments of the present application. As shown in Figure 3, the terminal device 100 includes a processor 111, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0078] The terminal device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0079] In some embodiments, the terminal device 100 may be a smart device. For example, the terminal device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0080] In some embodiments, as shown in FIG. 1 , the mobile terminal 300 or other smart display device can perform similar functions to the terminal device 100 after installing an application for operating the display device 200 .

[0081] The processor 111 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The processor 111 is used to control the operation and operation of the terminal device 100, as well as the communication and coordination between internal components and external and internal data processing functions.

[0082] The communication interface 130 communicates control signals and data signals with the display device 200 under the control of the processor 111. The communication interface 130 may include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, or other near field communication modules.

[0083] The user input / output interface 140 includes at least one of a microphone 141 , a touch panel 142 , a sensor 143 , a button 144 and other input interfaces.

[0084] In some embodiments, the terminal device 100 includes at least one of a communication interface 130 and an input / output interface 140. The terminal device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via the WiFi protocol, Bluetooth protocol, or NFC protocol and send them to the display device 200.

[0085] The memory 190 is used to store various operating programs, data, and applications that drive and control the terminal device 100 under the control of the processor. The memory 190 can store various control signal instructions input by the user.

[0086] The power supply 180 is used to provide operating power support for various components of the terminal device 100 under the control of the processor.

[0087] Figure 4 is a schematic diagram of the software configuration in the display device shown in Figure 1 provided in some embodiments of the present application. In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0088] In some embodiments, at least one application is running in the application layer. These applications may be window programs, system settings programs, clock programs, camera applications, etc. that come with the operating system; or they may be applications developed by third-party developers.

[0089] The framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes predefined functions and acts as a processing center, determining the actions taken by applications in the application layer.

[0090] As shown in FIG4 , in some embodiments of the present application, the application framework layer includes managers, content providers, and a view system, etc.

[0091] In some embodiments, the activity manager is used to manage the lifecycle of each application and general navigation back functionality.

[0092] In some embodiments, a window manager is used to manage all window programs.

[0093] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is accessed, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0094] In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG4 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, and sensor drivers (such as a fingerprint sensor, a temperature sensor, a touch sensor, a pressure sensor, etc.).

[0095] In some embodiments, the core layer further includes a power driver module for performing power management.

[0096] In some embodiments, the software programs and / or modules corresponding to the software architecture in FIG. 4 are stored in the first memory or the second memory shown in FIG. 2 or FIG. 3 .

[0097] In the related art of some embodiments, when a user is watching a video using a video player, the video being played may be configured with background music. For example, in the user interface shown in FIG5 , the emotional TV series XX is being played while the lyrical song YY is playing as background music. If the audio playing in the video is only a portion of the background music, the user may be interested in the background music and want to play other parts of the background music. The user can manually pause the video playback (for example, by clicking on the video screen to pause the video playback; in the user interface shown in FIG6 , the background music is also paused when the video is paused).

[0098] After that, the user obtains basic information about the background music (for example, by searching extensively online or asking other netizens in forums), and then searches the music software based on the basic information (in the user interface shown in FIG7 ) to hear the rest of the background music. After listening to the rest of the background music, the user returns to the video playback page and manually clicks the continue button of the video playback software to continue playing the video. This method of playing background music has many steps and is complicated, and the user experience is also poor.

[0099] To address the aforementioned issues in the related art, a method for media playback is provided according to an embodiment of the present application. The method for media playback according to an embodiment of the present application can be applied to the system shown in FIG8 . As shown in FIG8 , the system may include a server 400 and a display device 200 used by a user. For example, server 400 may be any data processing server, such as a cloud server or a distributed server.

[0100] In the user interface of the video player shown in FIG9 , in addition to displaying the video playback screen, a song identification button is also provided. When the user clicks the song identification button, the display device generates a song identification instruction in response to the click of the song identification button. In this case, the video player does not pause the currently playing video. After receiving the song identification instruction, the display device 200 begins recording the audio data of the currently playing video.

[0101] In some embodiments, a song identification button may not be set on the playback screen to avoid affecting the user's viewing experience. The user may trigger the display device to generate a song identification instruction by inputting a preset voice command or by using a preset case of a remote control device to enable the display device to generate a song identification instruction.

[0102] In some embodiments, when certain videos (e.g., TV series or movies) are played with background music, the audio of the characters in the video is often played simultaneously with the background music. To improve the accuracy of song identification, the audio data of the characters speaking can be removed from the recorded audio. The specific implementation of removing the audio data of the characters speaking is not described in detail in this application.

[0103] In some embodiments, while extracting a first audio segment of background music from the audio data of the current video playback, the first audio segment is sent to the server, so that the server searches for audio data of the background music based on the first audio segment.

[0104] In some embodiments, the present application can continuously record audio clips of background music, and continuously send newly recorded audio clips to the server, so that the server generates a combined clip for song identification after splicing the first audio clip after receiving the song identification request and the subsequent cumulatively received audio clips. Until the server 400 feeds back the audio data of the background music to the display device. Continuously recording audio clips of background music is uninterrupted recording, and continuously sending the recorded audio clips to the server 400 is periodically uploading the audio clips to the server 400. The period for uploading the audio clips to the server 400 can be gradually shortened. For example, the first period can be set to a longer period (1s) in order to provide as many spectral features as possible, the second period can be 50ms, and the third period can be 25ms... This can not only ensure sufficient spectral features to ensure recognition results, but also improve recognition efficiency. Therefore, the first audio clip in the embodiment of the present application refers to the audio clip sent by the display device 200 to the server 400 in the first period.

[0105] Exemplarily, when the display device does not receive the recognition result, it can continue to record the second audio clip, and in response to receiving the recognition result, upload the second audio clip to the server, and the server splices the first audio clip and the second audio clip to generate a combined clip for song identification; when the display device does not receive the recognition result, it can continue to record the third audio clip, and in response to receiving the recognition result, upload the third audio clip to the server, and the server splices the first audio clip, the second audio clip and the third audio clip to generate a combined clip for song identification; .... In this way, the collection cycle of the audio clip can be shortened, and the same collection time as in the cumulative related art cannot be applied to guarantee the success of song identification. Moreover, for some scenarios (for example, when the collected sound clip has more obvious spectral characteristics), song identification can be achieved without collecting the collection time in the related art.

[0106] In certain embodiments, audio data can be the audio content identified according to the accumulated sound fragments received. The duration threshold of the server 400 feedback audio data is set. If it exceeds the duration threshold, a timeout prompt is popped up on the user interface to prompt the user to currently search for an audio data supermarket. If no relevant music is searched, a prompt message that the relevant audio is not searched can also be popped up on the user interface. If the music player software currently installed by the display device 200 does not have the playback copyright for the background music, a prompt message that the playback copyright is not available can be popped up on the user interface. The embodiment of the present application is illustrative of the scheme with the fact that the server 400 retrieves audio data, the feedback audio data does not time out, and the music player software currently installed by the display device 200 has the playback copyright for the background music.

[0107] The specific process of searching for audio data is as follows: Server 400 establishes a communication relationship with display device 200, receives the first audio clip of the background music currently being played by the video player, and searches the database based on the first audio clip of the background music currently being played by the video player to obtain the music information of the background music currently being played by the video player. Server 400 then feeds back the music information of the background music currently being played by the video player to display device 200. The database in server 400 may be pre-configured and may store audio data of various background music and the corresponding background music information. The database may also store the background music information expected to correspond to audio data processed by an audio fingerprint algorithm.

[0108] If the server 400 cannot match the corresponding audio data in the database after receiving the first audio clip, the server 400 continues to receive the second audio clip uploaded by the display device 200. After splicing the first audio clip and the second audio clip, the server 400 generates a new audio clip (for example, an audio clip with a new audio fingerprint obtained by processing with an audio fingerprint algorithm), and then searches the database for audio data matching the fingerprint based on the new audio clip. If the corresponding audio data still cannot be matched in the database, the server 400 continues to receive the third audio clip uploaded by the display device 200. After splicing the first audio clip, the second audio clip and the third audio clip, the server 400 generates a new audio clip again, and then searches the database for audio data matching the fingerprint based on the new audio clip. If the corresponding audio data is matched in the database, the server 400 feeds back the audio data to the display device 200.

[0109] The song recognition button in the embodiment of the present application can be a control UI that is directly displayed on the video player application interface, or it can be a hidden control UI. If it is a hidden control UI, the hidden control UI will pop up and display only when the user calls out the hidden song recognition button through the remote control. For example, the song recognition button is not displayed on the video player application interface at the beginning. The user needs to click on the video player application interface first, and the song recognition button is displayed on the video player application interface. In order to avoid the user from accidentally clicking on the song recognition button, after the song recognition button is displayed on the video player application interface after the above operation, the focus does not immediately fall on the song recognition button, but the user needs to operate the remote control to move the focus to the song recognition button before clicking the song recognition button.

[0110] The control UI can be the player's playback UI, which can include a video player application interface for accessing background music and a UI for displaying the searched background music information. The background music entry control UI can be displayed as a graphical control, and the searched background music information results UI can be a sidebar added to the entry control UI graphic to display the searched background music information.

[0111] There are two ways to implement the above-mentioned acquisition of the first audio clip of the background music while playing the background music on the same display device 200:

[0112] In a first implementation, after an audio output device (e.g., a speaker) of the display device 200 outputs background music, an audio receiving device (e.g., a microphone) of the display device 200 receives the background music, and then an audio decoder of a video player on the display device 200 decodes the received background music to obtain a first audio clip of the background music.

[0113] The second implementation is that before the audio output device (such as a speaker) of the display device 200 outputs the background music, the display device 200 has decoded and obtained the first audio segment of the background music (for example, before the display device 200 plays the TV series, it obtains all the background music included in the TV series in advance and stores the audio data of all the background music in the memory). In this implementation, the display device 200 does not need to use an audio receiving device to receive the background music and decode it again to obtain the first audio segment. It only needs to read the first audio segment at the current moment from the memory after receiving the song recognition instruction. Compared with the first implementation, the second implementation can avoid the distortion and redundancy of the content of the received first audio segment due to external interference or failure of the audio receiving device during the secondary reception of background music. The second implementation can achieve accurate background music information even in a noisy environment. In addition, the second implementation can achieve accurate background music information even without an audio output device or an audio receiving device.

[0114] If the first audio segment is obtained in the first implementation manner, the audio receiving device continuously receives the background music without recording any time point.

[0115] If the second implementation method is used to obtain the first audio clip, the specific process may be:

[0116] After the display device 200 receives the song identification instruction, it generates a search time period with the time point of receiving the song identification instruction as the starting time point. A specific method of generating the search time period can be to obtain the end time point by adding a predetermined time length to the starting time point. The display device 200 decodes the background music of the time period based on the starting time point and the end time point to obtain the first audio clip of the background music being played by the video player. The display device 200 then sends the obtained first audio clip to the server 400, so that the server 400 searches for background music based on the first audio clip.

[0117] The above solution can be implemented by the core processor of the video player on the display device 200. In addition to being responsible for decoding the background music being played, the video player core can also be responsible for functions such as video reading (for example, downloading video data from the network), audio and video decoding, and audio and video synchronization. When the user triggers a function that requires searching for background music information, and the video player core decodes and obtains the first audio clip of the current background music, the video player core is also responsible for additionally transmitting the decoded audio data to the audio fingerprint calculation and nursing device used for fingerprinting the audio data to calculate the audio fingerprint.

[0118] The detailed process of the display device 200 acquiring the background music information of the background music being played is as follows:

[0119] An audio fingerprint calculation and processing device uses an audio fingerprint algorithm to perform fingerprint processing on a first audio segment of background music being played by a video player to generate an audio fingerprint corresponding to the first audio segment.

[0120] The audio fingerprint here can be an acoustic fingerprint, which can be a digital summary extracted from the audio signal of the background music through a specific algorithm, representing the digital signature of the background music in terms of acoustic feature content.

[0121] The audio fingerprint information is then sent to a third-party server via a network transmission device. A query is performed on the audio fingerprint database on the third-party server to obtain the background music information of the background music currently being played by the player. The database is used to store an audio fingerprint data table, which records pre-set audio fingerprint information and its corresponding background music information.

[0122] In this application, the above-mentioned audio fingerprint is mainly used to filter out similar or identical audio from the audio database and further obtain the background music information corresponding to the audio data. Here, the audio database stores the audio fingerprint of at least one audio data and its corresponding background music information (for example, the song title, composer, lyrics, playback address, download address, and singer of the background music). The audio fingerprint can be used as the index of the corresponding background music information in the database. The audio fingerprint calculation module must use the audio fingerprint algorithm used by the audio fingerprint stored in the database.

[0123] It should be noted that in the embodiment of the present application, the current video playing may be background music, or it may be opening music, ending music, interlude, etc. The embodiment of the present application uses background music as an example to illustrate the solution. If the current video playing is other sound data such as opening music, ending music, interlude, etc., the method of the embodiment of the present application can also be used to identify songs.

[0124] In the embodiment of the present application, the first audio clip specifically represents a clip of background music, while the audio data represents audio content similar to or identical to the background music. That is, the audio data may or may not be the same version as the background music. The audio data and the background music may have a high similarity in waveform spectrum or high similarity in sound text. For example, if the background music is song A sung by Li Si, the audio data may be song A sung by Li Si or song A sung by Wang Wu.

[0125] In practical applications of the background music search method of this application, the relationship between the searched audio data and the sound data may exist in the following four situations:

[0126] In the principle diagram of the relationship between the sound data of the first audio clip and the audio data as shown in Figure 10, the first case is that the sound data and the audio data are completely different (completely different here can refer to the completely different scope of the audio content), that is, there is no inclusion relationship or intersection relationship between the sound data and the audio data, then the server 400 feeds back the audio data to the display device 200, and the audio player on the display device 200 plays the audio data.

[0127] In the principle diagram of the relationship between sound data and audio data as shown in Figure 11, the second case is that if the audio data and the sound data are exactly the same, then the server 400 does not need to return the audio data to the display device 200, and the user can hear the complete version of the background music when watching the currently playing video. It is also possible that the currently existing resources are not complete versions, such as songs from a long time ago, so the complete version referred to in the embodiment of the present application refers to the most complete version that can be searched, rather than the original complete version of the background music. The user interface as shown in Figure 12 pops up the first prompt message in the user interface, prompting the user that the audio data currently found are identical to the sound data. At this time, since there is no need to open the audio player to play the audio data, there is no need to control the video player to pause the currently playing video.

[0128] In the third case, as shown in the diagram of the relationship between the sound data of the first audio segment and the audio data in FIG13 , the sound data and the audio data overlap. The server 400 feeds the audio data back to the display device 200 , and the audio player on the display device 200 plays the audio data.

[0129] In the principle diagram of the relationship between sound data and audio data shown in Figure 14, the fourth case is that the audio data includes sound data, then the server 400 feeds the audio data back to the display device 200, and the audio player on the display device 200 plays the audio data.

[0130] The signaling diagram shown in Figure 15 includes the following processes:

[0131] S1501: The user sends a song identification instruction to the display device 200;

[0132] S1502: The display device 200 sends a first audio clip to the server 400;

[0133] S1503: The server 400 feeds back the audio data to the display device 200;

[0134] S1504: The audio player on the display device 200 plays the audio data while the display device 200 controls the video player to pause playing the current video, and records the first playing progress of the current video when the current video is paused.

[0135] S1505 : The user sends an audio stop playback instruction to the display device 200 .

[0136] S1506, if the audio data is played to completion or the user clicks the stop play button on the audio player, the processor may receive an audio stop play instruction (at this moment, the audio stop play instruction may be automatically triggered and generated by the audio player or may be generated by receiving a click operation from the user). If the audio stop play instruction is automatically triggered and generated by the audio player due to the completion of the audio data playing, the audio player automatically stops playing the audio data. If the audio stop play instruction is generated due to receiving a click operation from the user, the processor needs to control the audio player to stop playing the audio data according to the audio stop play instruction. After the audio player stops playing the audio data, the processor feeds back the playback result (i.e., stops playing the audio data). The processor controls the video player to continue playing the current video based on the first playback progress according to the playback result.

[0137] For example, when a user is watching TV series XX through a video player (video progress is 25 minutes), background music YY is played. The user clicks the "Listen to Song and Identify Song" button in the video player page (video progress is 25 minutes and 10 seconds), and the display device 200 sends the first audio clip of background music YY (a clip of background music YY) to the server. The server 400 feeds back the audio data of background music YY (which can be the full version of background music YY) to the display device 200 at the 25th minute and 15 seconds of the video progress based on the first audio clip of background music YY. At the same time, the playback progress of TV series XX is paused at the 25th minute and 15 seconds of the video progress. The display device 200 starts to play the audio data of background music YY on the audio player. After the audio data of background music YY is played, the audio player is controlled to close, and the video player continues to play TV series XX from the 25th minute and 16 seconds of the video progress (at this time, TV series XX may still sound the clip of background music YY that has not been played, that is, the sound data has not been played).

[0138] In some embodiments, after receiving the audio data sent by the server 400, the display device 200 needs to decode the audio data. After the decoding is completed, the display device 200 pulls up the audio player, and then the display device 200 controls the video player to pause playing the current video.

[0139] In some embodiments, before the display device 200 controls the video player to pause playing the current video, if the user interrupts the song recognition operation, for example, the user presses the return button on the remote control or clicks the cancel control on the song recognition application, the display device 200 responds to the return instruction or the cancel instruction, and does not control the video player to pause playing the current video. At the same time, even if it receives audio data feedback from the server, it does not display anything related to song recognition on the page, but continues to play the current video.

[0140] According to the media resource playback method of the embodiment of the present application, the processor obtains the first audio clip of the background music of the current video playback in response to the song identification instruction input by the user, wherein the first audio clip is a clip of the background music. And the display device sends the first audio clip to the server. The server searches for the audio data of the background music based on the first audio clip. The audio data is a clip or a complete version of the background music, and the audio data includes at least the first audio clip. If the audio data also includes other audio data, the audio data fed back by the server is received, and the processor controls the video player to pause the current video, records the first playback progress of the current video, and controls the audio player to play the audio data. In response to the audio stop playback instruction input by the user, the audio player is controlled to stop playing the audio data, and at the same time, the video player is controlled to continue playing the current video based on the first playback progress. In this way, if the user is interested in the background music and is willing to play other parts of the background music, the user only needs to input the song identification instruction to achieve the purpose of audio and video linkage, thereby improving the user experience.

[0141] In some embodiments, after the server feeds back the audio data, the server can wait for the current video to finish playing the current background music before pausing the current video. Furthermore, the position of the current video playing the partial segment of the current background music can be recorded. After the current video finishes playing the background music segment, the audio player can be controlled to continue playing the partial segment of the background music instead of playing the full version of the background music.

[0142] The specific implementation process of the audio-visual linkage of this embodiment may be shown in the signaling diagram of FIG16 , including the following steps:

[0143] S1601. The user enters a song identification instruction by clicking the song identification button in the video playback application page, and sends the song identification instruction to the song identification application (the user can also directly click the song identification application, in which case the video player still plays the current video normally), where the song identification application can be an audio playback application with a song identification function.

[0144] S1602: The song recognition application determines the service APP currently in the playback state, such as the video playback application of this application. The song recognition application sends a communication establishment request to the video playback application. After the song recognition application and the video playback application establish the communication request, the song recognition application and the video playback application interact with each other.

[0145] S1603: The video player returns the playback progress information of the background music to the music recognition application (for example, the position of the specific audio clip of the background music currently being played in the video).

[0146] S1604: The music recognition application requests an audio file from the audio output module (the audio output module may be an audio player, and the audio file is the audio data of the background music being played, and may also include playback information of the audio data);

[0147] S1605. The audio output module returns the audio file to the song recognition application.

[0148] S1606. After receiving the audio file, the song recognition application continues to transmit the sound clips in the audio file to the song recognition cloud until it times out or returns the recognition result.

[0149] S1607. The song recognition cloud returns recognition information to the song recognition application.

[0150] S1608. After receiving the returned identification information, the song recognition application waits for the background music clip of the current video to finish playing, and searches for the corresponding song in the song recognition application according to the identification information.

[0151] S1609. At the same time, the music recognition application notifies the video player to pause video playback (the background music segment has been played) according to the playback progress information of the background music (that is, the current video has finished playing the background music segment).

[0152] S1610: After the video player stops playing the video, it feeds back a video progress interruption message to the song recognition application;

[0153] S1611. After receiving the video progress interruption information, the song identification application starts to play the searched song according to the background music playback progress information (it can be the full version or the progress of the background music in the current video playback). When the song identification application is playing the searched song, if it receives an audio stop playback instruction input by the user, the song identification application stops playing the searched song in response to the instruction. At this time, if the user inputs an instruction to exit the song identification application, the song identification application is exited in response to the instruction, and the video player continues to play the current video according to the recorded video progress interruption information.

[0154] S1612. If the searched song is finished playing (not considering the situation where the song recognition application automatically jumps to play other songs after playing the searched song), the user does not need to enter an instruction to exit the song recognition application. The song recognition application automatically sends an audio stop playback instruction to the video player. After receiving the instruction, the video player continues to play the current video according to the video progress interruption information.

[0155] For example, when a user is watching TV series XX through a video player (the video progress is at the 25th minute), background music YY is played. The user clicks the "Listen to Song and Identify Song" button on the video player page (the video progress is at the 25th minute and 10 seconds), and the display device 200 sends the first audio clip of background music YY (a clip of background music YY) to the server. The server 400 feeds back the audio data of background music YY (which can be the complete version of background music YY) to the display device 200 based on the first audio clip of background music YY at the 25th minute and 15 seconds of the video progress. The display device 200 waits for the first audio clip of background music YY to be played after completion (the video progress is at the 25th minute and 30 seconds) before pausing the playback of TV series XX, that is, the playback progress of TV series XX is paused at the 25th minute and 30 seconds of the video progress. The display device 200 starts playing the audio data of background music YY on the audio player. After playing the audio data of background music YY, the audio player is controlled to close, and the video player continues to play TV series XX from the 25th minute and 31 seconds of the video progress (at this time, since the first audio clip of background music YY has been played, the background music YY clip will no longer be played when playing TV series XX).

[0156] It should be noted that the server used for song identification and the server used for song search in this application can be different servers or the same server. The song identification cloud given in the above embodiment is a server for song identification, and the server for song search can also be a song identification cloud. For example, third-party music software usually has song identification and song search functions, so the song identification server and the song search server can both be third-party music software servers. The song identification server can also be a local server, and the local server is preset with a database for song identification. For example, the database mentioned above stores an audio fingerprint data table, which is used to record pre-set audio fingerprint information and its corresponding background music information.

[0157] In some embodiments, if the audio content of the sound data does not correspond to the end of the audio data, audio content similar to or identical to the first audio segment in the audio data is intercepted, and the audio data is played starting from the end point of the intercepted audio content. If the first audio segment is the end of the audio data, the audio data may be played starting from the starting point of the audio data.

[0158] For example, when a user is watching TV series XX through a video player, background music YY is played. The user clicks the "Listen to Song and Identify Song" button on the video player page, and the display device 200 sends the first audio clip of background music YY to the server. The server 400 feeds back the audio data of background music YY (the complete version of the background music) to the display device based on the first audio clip of background music YY. The total length of background music YY is 3 minutes and 20 seconds. If the first audio clip is the last 20 seconds, the audio player starts playing background music YY from the starting point of the background music, that is, from 0:01. If the first audio clip is a clip from 1 minute 10 seconds to 1 minute 30 seconds, the audio player starts playing the audio data from 1 minute 31 seconds.

[0159] In some embodiments, the display device 200 further receives music information of the background music fed back by the server 400, and controls the display of the music information on the display 200. If the music information includes lyrics of the background music, the lyrics corresponding to the first audio clip are controlled to be highlighted.

[0160] For example, the display device 200 also receives music information of background music fed back by the server 400, which includes the song title, lyrics, background image, etc. When a user is watching TV series XX through a video player and background music YY plays, the user interface shown in FIG17 is ultimately displayed, where the lyrics of the segment of background music YY used in TV series XX, "Tears Frozen on the Young Face," are magnified compared to the lyrics of other segments (in addition to magnification, highlighting, darkening, and other prominent display methods can also be used). This allows the user to distinguish between the segment used in the TV series and other segments, thereby enhancing the user's perception of the background music and improving the user experience.

[0161] In some embodiments, the character image in the current video screen is identified, and the music information is controlled to be displayed on the current video screen in a manner that does not block the character image. In the user interface shown in Figure 18, although the currently playing TV series XX is paused, the video screen is still displayed in the interface. At the same time, the music information (background picture, title, lyrics, etc.) of the background music YY is displayed below the video screen so that the music information does not block the character image display. Since background music is usually associated with the plot of the TV series, it has the effect of enhancing the plot. Therefore, the user can continue to watch the video screen while enjoying the background music, thereby further enhancing the user's perception of the background music and improving the user experience.

[0162] In certain embodiments, the condition that the audio player plays the audio data further comprises that the audio player supports the format of the audio data. If the audio player does not support the format of the audio data, a second prompt message is ejected on the display, wherein the second prompt message is used to prompt the user to install the audio player that supports playing the audio data.

[0163] In the user interface shown in Figure 19, after the display device 200 receives the full version of the background music YY fed back by the server 400, the audio player installed on the display device 200 does not support the full version of the background music YY, or no audio player is installed on the display device 200. Then a prompt message pops up on the user interface, prompting the user to install the corresponding audio player that supports playing the background music YY. A list of audio playback applications that can be installed can also be displayed on one side of the prompt message box that pops up. The audio playback application list is displayed by audio playback application 1, audio playback application 2, and audio playback application 3. The user can also click on the corresponding audio playback application control to directly install the application. For example, if the user clicks on the control of audio playback application 1, audio playback application 1 is directly installed on the display device 200. After the installation of audio playback application 1 is successful, background music YY is then played through audio playback application 1.

[0164] The present application provides a method for playing media assets. FIG20 is a flow chart of a method for playing media assets according to an exemplary embodiment. The method for playing media assets is applicable to the display device 200 in the system shown in FIG8 . As shown in FIG20 , the method for playing media assets may include the following steps:

[0165] In step S101, in response to the received song identification instruction, the first audio segment of the sound data of the current video playback is collected, and a song identification request containing the first audio segment is sent to the server, so that the server searches for the audio data of the sound data based on the first audio segment, wherein the first audio segment represents a segment of the sound data, and the audio data represents audio content that is similar to or identical to the sound data.

[0166] In step S102, if the audio content of the audio data exceeds the audio content range of the sound data, the audio data fed back by the server is received, the video player is controlled to pause playing the current video, the first playback progress of the current video when the current video is paused is recorded, and the audio player is controlled to play the audio data.

[0167] In step S103, in response to the received audio stop playback instruction, the audio player is controlled to stop playing the audio data, and the video player is controlled to continue playing the current video based on the first playback progress.

[0168] In some embodiments, if the audio content of the audio data does not exceed the audio content range of the sound data, the video player is not controlled to pause playing the current video.

[0169] In some embodiments, if the audio content of the sound data does not correspond to the end portion of the audio data, the audio data is played starting from the end point of an audio content that is similar to or identical to the audio content of the sound data. If the audio content of the sound data does not correspond to the end portion of the audio data, the audio data is played starting from the starting point of the audio data.

[0170] In some embodiments, the display device further receives music information of the background music fed back by the server and controls the display of the music information on the display. The music information includes lyrics of the background music, and the lyrics corresponding to the sound data are controlled to be highlighted.

[0171] In addition to the above-mentioned embodiments for solving the problem that the current method for playing background music for video is complicated in operation steps and leads to a poor user experience, the present application also provides an embodiment for enabling users to identify audio data of interest played by smart devices in application scenarios, as described below:

[0172] With the development of smart devices, multiple smart devices are increasingly being used in more and more application scenarios. For example, in a home internet scenario, display devices (such as smart TVs, smartphones, and tablets), smart speakers, refrigerators, air conditioners, and other home appliances may coexist. Display devices provide users with various entertainment functions such as audio, video, and games, while smart speakers can provide users with audio and other entertainment functions, meeting users' diverse entertainment needs.

[0173] If there is a smart device that can play audio in the application scenario, users may be interested in the audio played by the smart device and want to know relevant information about the audio. To meet this user demand, a method for audio recognition is urgently needed.

[0174] For example, in the scenario of home Internet, there are multiple home appliances that can play audio (such as TVs, mobile phones, tablets and other display devices, smart speakers), which can provide users with audio and other entertainment functions to meet users' different entertainment needs.

[0175] Among them, each display device can recognize audio by installing an application with audio recognition function. If a user is interested in the audio played by the display device or other home appliance and wants to know relevant information about the audio, according to the embodiment of the present application, a display device and audio recognition method can realize audio recognition in application scenarios with multiple devices to meet this user's needs.

[0176] To meet the above-mentioned needs of users, according to an embodiment of the present application, a display device and an audio recognition method can monitor the playback status of a first application running in the display device in response to a user's request to identify audio; after monitoring that the playback status is playing, based on the current time, the first audio data corresponding to the first application within a preset first time range is obtained, and the information of the first audio data can be identified by performing a corresponding audio recognition operation on the first audio data; after monitoring that the playback status is not playing, the second audio data within a preset second time range is obtained through a microphone (the sound source of the second audio data is a sound source outside the display device), and the information of the second audio data can be identified by performing a corresponding audio recognition operation on the second audio data. In an application scenario, the user is provided with the recognition of multiple audios in the application scenario, so that when the user encounters an intelligent device playing audio of interest or there is humming of interest in the environment, the user can identify the corresponding audio information, thereby improving the user experience.

[0177] FIG21 is a timing diagram of audio recognition in a display device according to some embodiments. As shown in FIG21 , the display device includes a microphone and a processor, wherein the processor is configured to perform the following steps:

[0178] S310: In response to a user request to recognize audio, monitor the playback status of a first application running on a display device.

[0179] The user's request for audio recognition may be issued by the user by clicking on the recognition control on the display device 200 , or by inputting by the user through buttons on the control apparatus 100 , voice input, and the like.

[0180] Figure 22 is a schematic diagram of a user's request for a display device to recognize audio in some embodiments. As shown in Figure 22, the display device has a recognition control 261, and the user operates the recognition control 261 through a control device, etc., to trigger a request to recognize audio.

[0181] In response to a user's request to identify audio, first, the playback status of a first application running in the display device is monitored, wherein the first application is an application with a playback function running in the display device, and the first application may be one or more.

[0182] If there are multiple first applications being played on the display device, a target application is determined from the multiple first applications based on a preset selection strategy, and first audio data corresponding to the target application is obtained.

[0183] For example, the first application being played includes a video playback application, a music playback application, and a game application, and all are playing audio. At this time, the preset selection strategy can be based on the type corresponding to the request for identifying audio. When the request for identifying audio is to identify a song, the target application corresponding to the corresponding preset selection strategy is the music playback application.

[0184] The playing status of the first application can be playing or not playing. By monitoring the playing status of the first application running on the display device, it is determined whether there is a sound being played by the first application on the display device.

[0185] If the playback status of the first application is monitored to be playing, step 320 is executed; if the playback status of the first application is monitored to be not playing, step 330 is executed.

[0186] S320: After monitoring that the playback status is playing, based on the current time, obtain the first audio data corresponding to the first application within a preset first time range, and identify the information of the first audio data by performing a corresponding audio recognition operation on the first audio data.

[0187] After monitoring that the playing state of the first application is playing, first audio data corresponding to a preset first time range is obtained based on the current time corresponding to the monitoring.

[0188] Among them, the first audio data is the data of the display device before audio decoding, that is, the data is not audio-decoded before the display device passes through the power amplifier and speaker. At this time, the data is a digital signal and only contains data corresponding to the audio information. Compared with the data after audio decoding of the display device, the data is purer and has less interference.

[0189] If the first audio data is data that the display device has not played after audio decoding, it needs to be encoded again. Alternatively, if the first audio data is data that the display device has played after audio decoding, that is, ambient sound recorded by a microphone, it may contain interfering sounds such as the user's voice, the sound of other devices, etc. Therefore, in the embodiment of the present application, the first audio data is the data of the display device before audio decoding, which improves the efficiency of audio data acquisition and the purity of the data.

[0190] Among them, by performing the corresponding audio recognition operation on the first audio data, the information of the first audio data is identified, and the audio data to be identified including the first audio data is passed to the recognition audio server, and according to the return result of the recognition audio server, the recognition result corresponding to the audio data to be identified is displayed in the result display area of ​​the display.

[0191] In some embodiments, after monitoring that the playing status of the first application is playing, the first audio data corresponding to the preset first time range is obtained without obtaining the sound recorded by the microphone to eliminate the interference of the microphone sound.

[0192] The information of the recognized first audio data may be displayed directly in the result display area, or may be displayed indirectly in the result display area through a QR code or the like; in some embodiments, it may also be displayed through voice information or the like.

[0193] For example, a video playback application in a display device is playing media A, and music B is playing in media A. At this time, the user is interested in music B, and can issue a user request to identify the audio by clicking on the identification control on the display device or by controlling buttons or voice input on the control device. In response to the user's request to identify the audio, and monitoring that the playback status of the display device is playing, the first audio data C within the preset first time range starting at this time is obtained. The first audio data C is part of music B, and the information of the first audio data C is identified by performing corresponding audio recognition operations on the first audio data C.

[0194] S330. After monitoring that the playback status is not played, obtain second audio data within a preset second time range through the microphone, and identify information of the second audio data by performing a corresponding audio recognition operation on the second audio data, where the sound source of the second audio data is a sound source outside the display device.

[0195] It should be understood that the display device is in a powered-on but not playing state, that is, the playing state of the first application running in the display device is not playing; but in the scene where the display device is located, there are other display devices or other smart devices outside the display device that are in a playing state, or there is audio of interest such as humming in the scene. At this time, the display device obtains the second audio data within the preset second time range through the microphone.

[0196] The preset second time is a time of the same length as or different length from the preset first time.

[0197] The second audio data is audio data outside the display device obtained through a microphone, and the sound source of the second audio data can be other display devices or other smart speakers outside the display device, or it can be other people in the scene humming without using a microphone, speaker or other device.

[0198] Figure 23 is a schematic diagram of an application scenario in which a display device is located according to some embodiments. As shown in Figure 23, the application scenario includes a display device 200, other display devices other than the display device (tablet 291, smart phone 292), a smart microphone 294, a smart speaker 293, people, animals, etc.; and the devices, people, and animals in the scenario can all be sound sources, and multiple audios in the scene are recognized through the display device 200.

[0199] It should be understood that the second audio data is obtained through a microphone and is determined after audio encoding and audio decoding of a sound source outside the display device.

[0200] The audio data outside the display device (i.e., the initial audio data) contains environmental noise (such as human voices, sounds of other devices, etc.). In order to make the obtained second audio data easy to identify, in some embodiments, the initial audio data can also be filtered through a preset audio filtering strategy to filter out the noise in the initial audio data and obtain the second audio data.

[0201] For example, if audio recognition takes the identification of songs (song recognition) as an example, at this time, the sound sources outside the display device include song M played by the smart speaker, the sound of car horns in the scene, and media N played by other smart devices (and no music). At this time, the sound of car horns and media N is filtered out through the preset audio filtering strategy, and the second audio data obtained only includes song M.

[0202] Among them, by performing the corresponding audio recognition operation on the second audio data, the information of the second audio data is identified, and the audio data to be identified including the second audio data is passed to the recognition audio server, and according to the return result of the recognition audio server, the recognition result corresponding to the audio data to be identified is displayed in the result display area of ​​the display.

[0203] It should be understood that the audio recognition in the embodiments of the present application includes aspects such as recognizing tracks, recognizing sounds, and recognizing audio information.

[0204] The display device performs steps through the processor to provide users with identification of audio played by the display device itself or other audio outside the display device in the application scenario, so that when the user encounters audio of interest played by the smart device or humming of interest in the environment, the user can identify the corresponding audio information, thereby improving the user experience.

[0205] In some embodiments, a first application on a display device is in a playing state. At this time, other audio data (e.g., audio data played by other smart devices or other people's humming) exists outside the display device. At this time, before performing a corresponding audio recognition operation on the first audio data in step 320 to identify information about the first audio data, the process may also include acquiring and analyzing audio data outside the display device. FIG24 is a timing diagram of audio recognition by a display device according to some embodiments. As shown in FIG24 , the processor in the display device is configured to perform the following steps:

[0206] S410: In response to a user request to recognize audio, monitor the playback status of a first application running on a display device.

[0207] The playing status of the first application can be playing or not playing. By monitoring the playing status of the first application running on the display device, it is determined whether there is a sound being played by the first application on the display device.

[0208] If the playback status of the first application is monitored to be playing, step 421 to step 424 are executed. If the playback status of the first application is monitored to be not playing, step 430 is executed.

[0209] S421: After monitoring that the playing state is playing, based on the current time, obtain first audio data corresponding to the first application within a preset first time range.

[0210] The first application on the display device is in a playing state. At this time, other audio data exists outside the display device, such as audio data played by other smart devices or other people humming. While obtaining the first audio data corresponding to the first application within a preset first time range based on the current time, step 422 may be continued.

[0211] S422: Monitor, through a microphone, whether there is audio data emitted by a sound source other than the display device within a preset first time range.

[0212] It should be understood that other sound sources besides the display device may be other smart devices, display devices, or human humming. For example, other sound sources besides the display device may be emitted by a mobile phone, a smart speaker, or other people or animals in the scene.

[0213] The microphone of the display device is used to monitor whether there is audio data emitted by other sound sources other than the display device within a preset first time range. If there is audio data emitted by other sound sources other than the display device, step 423 is executed.

[0214] S423: After monitoring the audio data emitted by other sound sources other than the display device, collect third audio data emitted by other sound sources through the microphone.

[0215] It should be understood that the third audio data, like the second audio data, is audio data outside the display device obtained by the display device through a microphone. Therefore, the third audio data is data obtained after audio encoding and audio decoding of the sound source outside the display device.

[0216] The third audio data is collected within a preset first time corresponding to the first audio data, and the acquisition time corresponding to the third audio data may be equal to or less than the preset first time.

[0217] For example, taking the display device as a smart TV, the media resources of the smart TV play music accompaniment, the user is interested in it, and identifies the song through the corresponding controls. At this time, the user hums the lyrics corresponding to the accompaniment through the smart microphone connected to the smart speaker. At this time, the smart TV obtains the music accompaniment (that is, the first audio data), and collects the lyrics corresponding to the accompaniment hummed by the user through the smart microphone through the microphone of the smart TV (that is, the third audio data); at this time, if audio recognition can be performed together with the music accompaniment and the corresponding lyrics, the efficiency of audio recognition can be improved.

[0218] In some embodiments, the acquisition of the first audio data and the third audio data can be performed simultaneously or in a sequential order. Since the methods of acquiring the first audio data and the third audio data are different, the efficiency of acquiring the first audio data and the third audio data can be improved by executing them simultaneously.

[0219] S424: Identify information of the audio data to be identified by performing a corresponding audio recognition operation on the audio data to be identified including the first audio data and the third audio data.

[0220] Among them, performing corresponding audio recognition operations through the first audio data and the third audio data to identify information of the audio data to be identified is to pass the audio data to be identified including the first audio data and the third audio data to the recognition audio server, and displaying the recognition result corresponding to the audio data to be identified in the result display area of ​​the display according to the return result of the recognition audio server.

[0221] When the display device simultaneously obtains the first audio data played within the display device and the third audio data outside the display device, it can determine whether the audio data to be identified includes the first audio data and the third audio data, or only includes the first audio data, or only includes the third audio data by judging the first audio data and the third audio data.

[0222] In some embodiments, the determination of the first audio data and the third audio data may include determining whether the third audio data corresponds to the first audio data, or may determine, based on the type of the identified audio, the content included in the third audio data and the first audio data that belongs to the audio data to be identified.

[0223] FIG25 is a flowchart of determining the first audio data and the third audio data according to some embodiments. As shown in FIG25 , in step 424, the following steps are also included:

[0224] S4241: Determine whether the third audio data corresponds to the first audio data.

[0225] S4242: If the third audio data corresponds to the first audio data, identify information of the first audio data and the third audio data by performing corresponding audio recognition operations on the first audio data and the third audio data.

[0226] It should be understood that the consistent correspondence between the third audio data and the first audio data indicates that the third audio data and the first audio data are different embodiments of the same data.

[0227] For example, the smart TV obtains the music accompaniment (i.e., the first audio data), and collects the lyrics corresponding to the accompaniment hummed by the user through the smart microphone through the microphone of the smart TV (i.e., the third audio data), or the smart TV obtains the music accompaniment (i.e., the first audio data), and collects the lyrics corresponding to the accompaniment hummed by the user through the microphone of the smart TV (i.e., the third audio data). At this time, it can be determined that the third audio data corresponds to the first audio data.

[0228] Among them, by performing corresponding audio recognition operations on the first audio data and the third audio data, the information of the first audio data and the third audio data is identified, and the audio data to be identified including the first audio data and the third audio data is passed to the recognition audio server, and according to the return result of the recognition audio server, the recognition result corresponding to the audio data to be identified is displayed in the result display area of ​​the display.

[0229] It should be noted that the audio data to be identified, including the first audio data and the third audio data, can be formed by combining the first audio data and the third audio data through a preset audio data combining strategy.

[0230] For example, the first audio data and the third audio data are combined according to the time of collection. When the third audio data corresponds to the first audio data, the first audio data and the third audio data corresponding to each collection moment are also consistent. Therefore, the first audio data and the third audio data can be combined according to the time of collection.

[0231] The first audio data and the third audio data are collected at the same time, so that the recognition of the audio data to be recognized that is a combination of the first audio data and the third audio data is more accurate.

[0232] S4243: If the third audio data does not correspond to the first audio data, identify information of the first audio data by performing a corresponding audio recognition operation on the first audio data.

[0233] If the third audio data does not correspond to the first audio data, the first audio data on the display device is used as the basis for audio recognition by default.

[0234] Among them, by performing the corresponding audio recognition operation on the first audio data, the information of the first audio data is identified, and the audio data to be identified including the first audio data is passed to the recognition audio server, and according to the return result of the recognition audio server, the recognition result corresponding to the audio data to be identified is displayed in the result display area of ​​the display.

[0235] In some embodiments, the first audio data and the third audio data can be analyzed based on the type of audio recognition request to determine the audio data to be recognized, and a corresponding audio recognition operation can be performed on the audio data to be recognized to identify information of the audio data to be recognized; wherein, the audio data to be recognized includes the first audio data, or the third audio data, or the first audio data and the third audio data.

[0236] For example, audio recognition is the recognition of animal sounds. The first audio data is the audio data played by the display device media, which is the conversation between the characters in the media. The third audio data is a bird sound played by a smartphone. At this time, the third audio data is inconsistent with the first audio data. The audio recognition request performs the corresponding audio recognition operation on the third audio data to identify the information of the third audio data.

[0237] Among them, by performing the corresponding audio recognition operation on the third audio data, the information of the third audio data is identified, and the audio data to be identified including the third audio data is passed to the recognition audio server, and according to the return result of the recognition audio server, the recognition result corresponding to the audio data to be identified is displayed in the result display area of ​​the display.

[0238] S430. After monitoring that the playback status is not played, obtain second audio data within a preset second time range through the microphone, and identify information of the second audio data by performing a corresponding audio recognition operation on the second audio data, where the sound source of the second audio data is a sound source outside the display device.

[0239] It should be understood that the display device is in a powered-on but not playing state, that is, the playing state of the first application running in the display device is not playing; but in the scene where the display device is located, there are other display devices or other smart devices outside the display device that are in a playing state, or there is audio of interest such as humming in the scene. At this time, the display device obtains the second audio data within the preset second time range through the microphone.

[0240] The preset second time is a time of the same length as or different length from the preset first time.

[0241] The second audio data is audio data outside the display device obtained through a microphone, and the sound source of the second audio data can be other display devices outside the display device or other smart device speakers played, or it can be other people in the scene humming without using microphones, speakers or other devices.

[0242] It should be understood that the second audio data is obtained through a microphone and is determined after audio encoding and audio decoding of a sound source outside the display device.

[0243] The audio data outside the display device (i.e., the initial audio data) contains environmental noise (such as human voices, sounds of other devices, etc.). In order to make the obtained second audio data easy to identify, in some embodiments, the initial audio data can also be filtered through a preset audio filtering strategy to filter out the noise in the initial audio data and obtain the second audio data.

[0244] Among them, by performing the corresponding audio recognition operation on the second audio data, the information of the second audio data is identified, and the audio data to be identified including the second audio data is passed to the recognition audio server, and according to the return result of the recognition audio server, the recognition result corresponding to the audio data to be identified is displayed in the result display area of ​​the display.

[0245] The display device performs steps through the processor to provide users with identification of audio played by the display device itself or other audio outside the display device in the application scenario, so that when the user encounters audio of interest played by the smart device or humming of interest in the environment, the user can identify the corresponding audio information, thereby improving the user experience.

[0246] In some embodiments, a first application on a display device is in a "playing" state, where the "playing" state may include multiple play states. After monitoring that the play state is "playing" in step 320, a recording strategy corresponding to the first audio data may be further determined based on the multiple play states. FIG26 is a timing diagram of audio recognition by a display device in some embodiments. As shown in FIG26 , a processor in the display device is configured to execute the following steps:

[0247] S510: In response to a user request to recognize audio, monitor a playback status of a first application running on a display device.

[0248] If the playback status of the first application is monitored to be playing, step 521 to step 522 are executed; if the playback status of the first application is monitored to be not playing, step 530 is executed.

[0249] S521: After monitoring that the playing state is playing, determine the recording strategy corresponding to the playing state of the first application program.

[0250] When the playing state of the first application program being played includes a first playing state and a second playing state, determining the playing state of the first application program being played includes the following steps:

[0251] If the playback state of the first application is the first playback state, based on the preset recording strategy, it is determined to obtain a first recording strategy corresponding to the first playback state.

[0252] If the play state of the first application is the second play state, based on the preset recording strategy, it is determined to obtain the second recording strategy corresponding to the first play state.

[0253] For example, smart TVs include two types of audio according to business scenarios: ordinary media playback (such as the playback of movies and TV series, which has low latency requirements and high sound effect requirements) and call and karaoke vocal playback (requires low latency and low sound quality requirements). The links of these two types of media playback are different. Media playback uses ordinary playback channels, and karaoke vocal playback uses low-latency channels. The recording nodes are also different for different channels. At this time, for media applications, the corresponding playback status of the media application is the first playback status; for karaoke vocal applications, the corresponding playback status of the media application is the second playback status. Based on the preset recording strategy, it is determined that the media application corresponds to the first recording strategy, and the karaoke vocal application corresponds to the second recording strategy.

[0254] S522. Based on the current time and the recording strategy, obtain first audio data corresponding to the first application within a preset first time range, and identify information of the audio data to be identified by performing a corresponding audio recognition operation on the audio data to be identified that includes the first audio data.

[0255] The acquisition of the first audio data also needs to be implemented according to a corresponding recording strategy.

[0256] In some embodiments, the principles of step 522 and step 320 are the same and will not be repeated here.

[0257] In some embodiments, in step 522, in the process of obtaining the first audio data corresponding to the first application within a preset first time range based on the current time, there is other audio data in addition to the display device, such as audio data played by other smart devices or humming by other people.

[0258] First, the microphone is used to monitor whether there is audio data emitted by other sound sources other than the display device within a preset first time range.

[0259] It should be understood that other sound sources besides the display device may be other smart devices, display devices, or human humming. For example, other sound sources besides the display device may be emitted by a mobile phone, a smart speaker, or other people or animals in the scene.

[0260] Secondly, after monitoring the audio data emitted by other sound sources other than the display device, the third audio data emitted by the other sound sources is collected through the microphone.

[0261] The third audio data is the data obtained by audio encoding and audio decoding of a sound source outside the display device. The third audio data is collected within a preset first time corresponding to the first audio data. The acquisition time corresponding to the third audio data can be equal to or less than the preset first time.

[0262] In some embodiments, the acquisition of the first audio data and the third audio data can be performed simultaneously or in a sequential order. Since the methods of acquiring the first audio data and the third audio data are different, the efficiency of acquiring the first audio data and the third audio data can be improved by executing them simultaneously.

[0263] Finally, by performing a corresponding audio recognition operation on the audio data to be recognized including the first audio data and the third audio data, information of the audio data to be recognized is recognized.

[0264] When the display device simultaneously obtains the first audio data played within the display device and the third audio data outside the display device, it can determine whether the audio data to be identified includes the first audio data and the third audio data, or only includes the first audio data, or only includes the third audio data by judging the first audio data and the third audio data.

[0265] In some embodiments, the determination of the first audio data and the third audio data may include determining whether the third audio data corresponds to the first audio data, or may determine, based on the type of the identified audio, the content included in the third audio data and the first audio data that belongs to the audio data to be identified.

[0266] If the third audio data corresponds to the first audio data, corresponding audio recognition operations are performed on the first audio data and the third audio data to identify information about the first audio data and the third audio data. If the third audio data does not correspond to the first audio data, corresponding audio recognition operations are performed on the first audio data to identify information about the first audio data.

[0267] S530. After monitoring that the playback status is not played, obtain second audio data within a preset second time range through the microphone, and identify information of the second audio data by performing a corresponding audio recognition operation on the second audio data, where the sound source of the second audio data is a sound source outside the display device.

[0268] The display device performs steps through the processor to provide users with identification of audio played by the display device itself or other audio outside the display device in the application scenario, so that when the user encounters audio of interest played by the smart device or humming of interest in the environment, the user can identify the corresponding audio information, thereby improving the user experience.

[0269] FIG27 is a timing diagram of an audio recognition operation performed by a display device through an audio recognition server according to some embodiments. As shown in FIG27 , the audio data to be recognized may be determined based on different audio data, specifically including the following steps:

[0270] S610: Determine audio data to be recognized.

[0271] Based on the timing diagrams of FIG. 5 to FIG. 26 , it should be understood that the audio data to be recognized includes: the first audio data, or the second audio data, or the third audio data, or the first audio data and the third audio data.

[0272] S620: Deliver the audio data to be recognized to the recognition audio server.

[0273] On the audio recognition server side, recognition is performed on the audio data to be recognized.

[0274] In some embodiments, the type of audio data to be identified can be reduced to a preset data type on the identification audio server. After the type is reduced, the amount of audio data to be identified is smaller, the speed of transmission to the identification audio server is faster, and the efficiency of identification by the identification audio server is also improved. Therefore, before step 520, the type of audio data to be identified can be reduced to a preset data type on the identification audio server.

[0275] S630: Receive the recognition result sent by the audio recognition server, and display the recognition result on a display device.

[0276] The information of the recognized audio data to be recognized (recognition results) can be displayed directly in the result display area, or indirectly in the result display area through a QR code or the like; in some embodiments, it can also be displayed through voice information or the like.

[0277] It should be understood that there are many types of audio data to be identified. Generally, the identification is achieved on the server side as shown in Figure 27. In some embodiments, the identification of the audio data to be identified can also be achieved through the local data of the display device.

[0278] In an embodiment of the present application, the display device obtains its own first audio data and / or the third audio data outside the display device at this time, and obtains the second audio data outside the display device when the display device does not have its own first audio data, and uses each audio data as audio data to be identified. The cloud-based server for identifying audio is used to recognize various audio data in the application scenario, thereby improving the user experience. At the same time, the efficiency of audio recognition can also be improved in some scenarios.

[0279] In addition to the above-mentioned embodiment of identifying audio-related information, in order to further improve the efficiency and accuracy of identifying audio-related information, this application also provides some other information identification embodiments, which are described in detail as follows:

[0280] The song recognition function can identify the audio data being played to obtain song information of the audio data, such as song title, artist, album, etc. Users can use the song recognition function of the display device to identify the song information and then play or download the corresponding audio data online based on the song information.

[0281] Display devices typically implement song recognition by installing an application with song recognition capabilities. When a user becomes interested in a song, they can launch the application to identify the song being played. The recognition process involves recording the playing song using the display device's microphone. The recorded audio data is then uploaded to a server, which compares it with audio data in a database to find a match. The matching song information is then sent to the display device for display.

[0282] The above song information recognition method only compares the recorded audio data with the audio data in the audio library, and the song information recognition efficiency and accuracy are low.

[0283] In some embodiments, the song recognition function can identify the audio data being played to obtain song information of the audio data, such as the song title, artist, album, etc. The user can use the song recognition function of the display device to identify the song information and then play or download the corresponding audio data online based on the song information.

[0284] Display devices typically implement song recognition by installing an application with song recognition capabilities. When a user becomes interested in a song they hear, they can launch the application with song recognition capabilities to identify the song being played. The specific recognition process involves recording the song being played through the display device's microphone. The recorded audio data is then uploaded to a server, which compares the recorded audio data with the audio data in a database to find audio data in the database that matches the recorded audio data. The song information of the matching audio data is then sent to the display device for display. The above-mentioned song information recognition method only uses the recorded audio data to compare with the audio data in the audio library, and the efficiency and accuracy of song information recognition are low.

[0285] To address the issues described in the above embodiments, a song information identification method is provided according to an embodiment of the present application. The song information identification method according to an embodiment of the present application can be applied to the system shown in FIG28 . As shown in FIG28 , the system may include: a server 400 and a display device 200 used by a user. The server 400 may be, for example, a cloud server, a distributed server, or any other form of data processing server.

[0286] In the user interface shown in FIG29 , in addition to displaying various controls, a song identification button is also provided. When a user clicks the song identification button, the display device generates a song identification instruction in response to the click of the song identification button. The currently playing sound data is not paused. After receiving the song identification instruction, the display device 200 begins recording the currently playing sound data.

[0287] In some embodiments, a song identification button may not be provided on the user interface to avoid affecting the user's viewing experience. The user may trigger the display device to generate a song identification instruction by inputting a preset voice command or by using a preset case of a remote control device to enable the display device to generate a song identification instruction.

[0288] In some embodiments, when certain videos (e.g., TV series or movies) are played with background music, the audio of the characters in the video is often played simultaneously with the background music. To improve the accuracy of song identification, the audio data of the characters speaking can be removed from the recorded audio. The specific implementation of removing the audio data of the characters speaking is not described in detail in this application.

[0289] The song recognition button in the embodiment of the present application can be a control UI that is directly displayed on the video player application interface, or it can be a hidden control UI. If it is a hidden control UI, the hidden control UI will pop up and display only when the user calls out the hidden song recognition button through the remote control. For example, the song recognition button is not displayed on the video player application interface at the beginning. The user needs to click on the video player application interface first, and the song recognition button is displayed on the video player application interface. In order to avoid the user from accidentally clicking on the song recognition button, after the song recognition button is displayed on the video player application interface after the above operation, the focus does not immediately fall on the song recognition button, but the user needs to operate the remote control to move the focus to the song recognition button before clicking the song recognition button.

[0290] The control UI may be a player's playback UI, which may include a video player application interface for accessing audio data and a UI for displaying the searched song information. The audio data entry control UI may be displayed as a graphical control, and the searched song information UI may include a sidebar added to the entry control UI graphic to display the searched song information.

[0291] There are two ways to obtain the audio segment of the sound data:

[0292] In a first implementation, after the audio output device (e.g., a speaker) of the display device 200 outputs the sound data, the audio receiving device (e.g., a microphone) of the display device 200 receives the sound data, and then the audio decoder of the video player on the display device 200 decodes the received sound data to obtain an audio segment of the sound data.

[0293] The second implementation is that before the audio output device (such as a speaker) of the display device 200 outputs the sound data, the display device 200 has decoded and obtained the audio segment of the sound data (for example, before the display device 200 plays the TV series, it obtains all the sound data included in the TV series in advance and stores the audio data of all the sound data in the memory). In this implementation, the display device 200 does not need to use an audio receiving device to receive the sound data and decode it again to obtain the audio segment. It only needs to read the audio segment at the current moment from the memory after receiving the song identification instruction. Compared with the first implementation, the second implementation can avoid the distortion and redundancy of the received audio segment content due to external interference or failure of the audio receiving device during the second reception of sound data. The second implementation can achieve accurate sound data information even in a noisy environment. In addition, the second implementation can achieve accurate sound data information even without an audio output device or an audio receiving device.

[0294] If the audio segment is obtained in the first implementation manner, the audio receiving device continuously receives the sound data without recording any time point.

[0295] If the second implementation method is used to obtain the audio clip, the specific process can be:

[0296] After the display device 200 receives the song identification instruction, it generates a search time period with the time point of receiving the song identification instruction as the starting time point. A specific method of generating the search time period can be to obtain the end time point by adding a predetermined time length to the starting time point. The display device 200 decodes the sound data of the time period based on the starting time point and the end time point to obtain an audio segment of the sound data currently being played by the video player. The display device 200 then sends the obtained audio segment to the server 400, so that the server 400 searches for the sound data based on the first audio segment.

[0297] The above solution can be implemented by the core processor of the player on the display device 200. In addition to decoding the played sound data, the player core can also be responsible for video reading (for example, downloading video data from the network), audio and video decoding, audio and video synchronization, etc. When the user triggers the function that requires searching for song information, and the player core decodes the audio clip of the current sound data, the player core is also responsible for transmitting the decoded audio data to the audio fingerprint calculation and nursing device for fingerprint processing of the audio data to calculate the audio fingerprint.

[0298] In some embodiments, if the song information of the sound data is identified only based on the audio clip, the detailed process of the display device 200 obtaining the song information of the sound data being played is as follows:

[0299] The audio fingerprint calculation and processing device uses an audio fingerprint algorithm to perform fingerprint processing on the audio segment of the sound data being played by the player to generate an audio fingerprint corresponding to the audio segment.

[0300] The audio fingerprint here can be an acoustic fingerprint, which can be a digital summary extracted from the audio signal of the background music through a specific algorithm, representing the digital signature of the background music in terms of acoustic feature content.

[0301] The audio fingerprint information is then transmitted to a third server via a network transmission device. A query is performed using the audio fingerprint database on the third server to obtain the song information of the sound data currently being played by the player. The database is used to store an audio fingerprint data table, which is used to record pre-set audio fingerprint information and its corresponding song information.

[0302] In this application, the above-mentioned audio fingerprint is mainly used to filter out similar or identical audio from the audio database and further obtain the song information corresponding to the audio data. The audio database here stores the audio fingerprint of at least one audio data and its corresponding song information (for example, song title, songwriter, lyrics, play address, download address and singer, etc.). The audio fingerprint can be used as the index of the corresponding song information in the database. The audio fingerprint algorithm used by the audio fingerprint calculation module must be consistent with the algorithm used by the audio fingerprint stored in the database.

[0303] It should be noted that the sound data currently being played in the embodiment of the present application can be the background music of a video or an audio novel, or it can be the opening song, ending song, interlude, complete song, etc.

[0304] The embodiment of the present application is an embodiment of combining an audio clip with the content in a control to identify song information, wherein the content in the control is specifically the media content corresponding to the control. The display device first displays the current user interface, and the display device 200 responds to the received song identification instruction, collects the audio clip of the sound data being played, and obtains the content data corresponding to the control in the current user interface. The display device 200 then sends the collected audio clip and content data to the server 400. The server 400 identifies the song information corresponding to the sound data based on the audio clip and content data, and feeds the song information back to the display device 200. The display device 200 receives and displays the identified song information fed back by the server 400. In this way, the song information of the sound data is identified by adding the content data of the control on the basis of the recorded audio data, which can improve the efficiency and accuracy of song information recognition.

[0305] The signaling diagram shown in FIG30 includes the following steps:

[0306] S3001: The user sends a song identification instruction to the display device 200;

[0307] S3001: The display device 200 collects an audio clip of the sound data being played;

[0308] S3003: The display device 200 determines whether there is an active control in the current user interface. If yes, step S3004 is executed; otherwise, step S3006 is executed.

[0309] S3004. If there is an active control in the current user interface, the processor obtains the first content data of the active control, wherein the first content data represents the media information corresponding to the active control, but does not represent the media information corresponding to the inactive control. An active control refers to a control with dynamic items, and an inactive control refers to a control without dynamic items. An active control may be a video playback control (such as a video player), an audio playback control (such as an audio player), a picture playback control (picture player), and other controls. The dynamic items in the video playback control may include items such as continuously played video frames, a progress bar, and scrolling lines. The dynamic items in the audio playback control may include items such as sound effect display and scrolling lyrics. The dynamic items of the picture playback control may include items such as scrolling pictures. For the above-mentioned active controls, the first content data obtained may be plain text and pictures.

[0310] In some embodiments, the presence of a dynamic item indicates that the audio and video content corresponding to the control is being played in the current interface. The audio and video content refers to audio and / or video content.

[0311] In some embodiments, the collected sound clip is the sound data played by the display device itself. The sound data played by itself is generally the sound content of the audio and video content corresponding to the existence of the control on the current interface. Therefore, the control title, introduction text, lines, and audio and video media title of the control corresponding to the active item can be used to more accurately and quickly determine the track corresponding to the sound clip.

[0312] In some embodiments, the title of the control refers to the text control title outside the image area of ​​the control. The audio and video media asset title can be the media asset title in the audio and video information corresponding to the media asset played by the player.

[0313] After acquiring the first content data of the active control, the audio clip and the first content data are sent to the server 400;

[0314] S3005: The server 400 identifies the first song information of the sound data according to the audio clip and the first content data, and then feeds back the first song information to the display device 200.

[0315] S3006. If no active controls exist in the current user interface, obtain second content data for inactive controls on the current page. Inactive controls refer to spaces without dynamic items. Inactive controls may include navigation bar controls, sidebar controls, and other controls. Items in inactive controls are generally static text (non-scrolling text) and single static images (non-scrolling images). For these inactive controls, the obtained second content data may also be plain text or images.

[0316] After acquiring the second content data of the inactive control, the audio clip and the second content data are sent to the server 400;

[0317] S3007: The server 400 identifies the second song information of the sound data according to the audio clip and the second content data, and then feeds back the second song information to the display device 200.

[0318] It should be noted that the first content data and the second content data are different data, and the first song information and the second song information are different information. Usually, the number of active controls in the user interface is much smaller than the number of inactive controls. Therefore, if there are active controls in the user interface, the song information is identified only based on the content data and audio clips of the active controls, which can improve the efficiency of song information identification. And the sound data usually comes from the active controls, for example, background music comes from a video player, and songs come from an audio player. Therefore, the song information is identified only based on the first content data and audio clips of the active controls, which can also improve the accuracy of song information identification. If there are no active controls in the user interface, the second content data and audio clips of the inactive controls can also be sent to the server 400. The server 400 combines the second content data and the audio clip to identify the song information. Compared with identifying the song information only through the audio clip, it can also improve the efficiency and accuracy of song information identification.

[0319] In some embodiments, the process of obtaining the first content data may include: before receiving the song identification instruction, the system pre-determining the active control in the current user interface. The system may determine whether the control in the user interface is the active control by taking screenshots of the user interface at preset time intervals, and determining whether the active control has changed based on the screenshots; if so, the control is the active control; if not, the control is the inactive control.

[0320] For example, the user interface shown in Figure 29 is the user interface of a video playback platform, which includes a navigation bar, a sidebar, a video recommendation bar, and a video playback bar. The navigation bar contains multiple text controls, the sidebar includes multiple text controls, the video recommendation bar includes text controls and image controls, and the video playback bar includes multiple video playback controls. The video is being played for the video playback control B, and the video playback control A and the video playback control C only display an introduction picture (not considering the situation of scrolling display pictures). For example, the user interface shown in Figure 29 can be screenshoted multiple times within 1 minute. Since only the image of the video playback control B is constantly changing in the user interface shown in Figure 31, it can be determined that the video playback control B is the active control. The text controls in the navigation bar, the text controls in the sidebar, the video playback control A, and the video playback control B are all inactive controls.

[0321] Then, when the user inputs a song recognition command, a screenshot of the current user interface is taken. After taking the screenshot, the screenshot can be cropped to retain only the active controls for content recognition. For example, in the user interface shown in FIG31 , a screenshot of the current user interface is taken and then cropped to retain only the video playback control B for content recognition. The method for text recognition on an image can be as follows:

[0322] First, the cropped image is binarized to obtain a binarized image. Binarization of the image can be to set the grayscale value of the pixel points on the image to 0 or 255, so that the entire image presents a black and white effect. Binarization is a method of image segmentation. When binarizing the image, the pixel grayscale greater than the critical grayscale value preset in the mobile terminal can be used as the grayscale maximum value (the grayscale maximum value can be 255), and the pixel grayscale less than the critical grayscale value can be used as the grayscale minimum value (the grayscale maximum value can be 0), thereby achieving image binarization. The binarization algorithm can use a global fixed threshold or a local adaptive threshold, which is not limited in the embodiment of the present invention. If the target image is a color image, it is necessary to grayscale the color image before binarizing the image to obtain a grayscale image, and then binarize the grayscale image to obtain a binary image.

[0323] Then, the black effect part of the binary image with black and white effect is framed. The black effect part is actually a collection of pixels. The mobile terminal can determine the distance between all adjacent pixels contained in the black effect part of the entire image, and frame the pixels contained in the black effect part of the entire image based on the distance. At least one target frame can be obtained, and the distance between adjacent pixels in the black part of each target frame is less than or equal to the preset pixel distance in a certain mobile terminal. After the black part of the binary image is framed and at least one target frame is obtained, the center point coordinates of each target frame and the area of ​​each target frame can be obtained respectively with the lower left corner of the target image as the origin of the plane rectangular coordinate system. A noise frame set and a text frame set are determined based on at least one target frame; wherein, the center point coordinate distance between adjacent target frames in the noise frame set is less than the preset distance, and the center point coordinate distance between adjacent target frames in the text frame set is greater than or equal to the preset distance. Obtain the total area of ​​the noise frame set and the total area of ​​the text frame set. The total area of ​​the noise frame set is the sum of the areas of all target frames contained in the noise frame set, and the total area of ​​the text frame set is the sum of the areas of all target frames contained in the text frame set. Perform text recognition on the text frame set to obtain the text information contained in the binary image.

[0324] According to the above embodiment, the detailed process of obtaining the content data of the activity control, combining the content data of the activity control with the audio clip, and performing song information recognition on the sound data being played is as follows:

[0325] An audio fingerprint calculation and processing device uses an audio fingerprint algorithm to fingerprint the audio clip of the sound data currently being played by a player, generating an audio fingerprint corresponding to the audio clip. This audio fingerprint information is then transmitted to a third-party server via a network transmission device. A query is performed on the audio fingerprint database on the third-party server to obtain the background music information currently being played by the player. The database stores an audio fingerprint data table, which records pre-set audio fingerprint information and its corresponding song information.

[0326] Similar or identical audio is filtered out from the audio database based on the generated audio fingerprint. If multiple similar or identical audios are filtered out from the audio database, then the multiple similar or identical audios are further filtered out based on the first content data (or second content data) to obtain the audio that finally matches the first content data (or second content data), that is, the final matching first song information is obtained.

[0327] For example, if the sound data playing is song A sung by Li Si, searching the audio database based on the audio fingerprint may simultaneously retrieve song information for song A sung by Li Si, song A sung by Wang Wu, and song A sung by Zhao San. Using conventional methods, accurate song information corresponding to the sound data cannot be obtained. According to the method of an embodiment of the present application, if an active control exists in the current user interface, the song information for song A sung by Li Si, song A sung by Wang Wu, and song A sung by Zhao San are further filtered using the first content data of the active control. For example, the lyrics for song A sung by Li Si may differ from those for song A sung by Wang Wu and song A sung by Zhao San. However, after text comparison, the lyrics for the song played by the sound data overlap with those for song A sung by Li Si. Therefore, it can be determined that the sound data playing is song A sung by Li Si. Finally, the server 400 feeds back the song information for song A sung by Li Si to the display device 200. The display device 200 then receives the song information for song A sung by Li Si and can display the song information for song A sung by Li Si on the user interface.

[0328] In some embodiments, if the active control is a video playback control, the video name is displayed on the video playback control, and the sound data is the sound data being played by the video playback control, the audio clip and the video name are sent to the server 400. The server 400 searches for a first configuration file corresponding to the first video with the video name, wherein the first configuration file includes at least all the audio data configured for the first video and the song information corresponding to the audio data. The first song information fed back by the server is received, wherein the first song information is the song information of the audio data matched with the audio clip found in the configuration file. The first configuration file includes the audio data and song information of songs such as the background song, opening song, interlude, and ending song configured for the first video.

[0329] For example, in the user interface shown in FIG32 , video player control B is playing a video and audio data simultaneously. The title of the TV series CCC is also displayed in the playback window of video player control B. After receiving a song identification command input by the user, display device 200 sends the title of the TV series CCC and the recorded audio clip to server 400. Server 400 first searches the audio database for songs 1, 2, and 3 based on the recorded audio clip. It then searches the configuration file corresponding to the video based on the title of the TV series CCC. For example, in the configuration file diagram shown in FIG33 , TV series CCC is configured with songs 2, 4, 5, etc. Songs 1, 2, and 3 are found in the audio database based on the recorded audio clip. These are then matched against the titles 2, 4, 5, etc. in the configuration file (the matching method here can be based on the text of the song titles or further based on the song information). The server concludes that the sound data currently playing is from song 2. Finally, server 400 returns the song information for song 2 to display device 200.

[0330] In some embodiments, if the active control is a video playback control, video lines are displayed on the video playback control, and the sound data is the sound data being played by the video playback control, the audio clip and the first video lines are sent to the server 400. If the server 400 can find a second configuration file for the second video based on the first video lines, where the second configuration file at least includes all audio data configured for the second video and song information corresponding to the audio data, the server 400 feeds back song information to the display device 200, where the song information is the song information of the audio data matching the audio clip found in the second configuration file.

[0331] For example, in the user interface shown in FIG34 , video player control B is playing a video and simultaneously playing sound data. The line "Shang Yi Zhang Hong" from the TV series is also displayed in the playback window of video player control B. After receiving a song identification command input by the user, display device 200 sends the line "Shang Yi Zhang Hong" from the TV series and the recorded audio clip to server 400. Server 400 first searches the audio database for songs 1, 2, and 3 based on the recorded audio clip. It then searches for the TV series CCC containing the line "Shang Yi Zhang Hong." It then searches the configuration file corresponding to the video based on the TV series name CCC. TV series CCC is configured with songs 2, 4, 5, and so on. Based on the recorded audio clip, it searches the audio database for songs 1, 2, and 3, matches them with the configuration files for songs 2, 4, 5, and so on, and concludes that the sound data currently playing is from song 2. Server 400 then sends the song information for song 2 back to display device 200.

[0332] In some embodiments, if the server 400 cannot find the second configuration file for the second video based on the first video's lines (essentially, the server 400 cannot determine a unique video based on the first video's lines; a line may correspond to multiple videos), that is, the display device 200 does not receive any song information from the server 400, the display device 200 will continue to send the second video's lines to the server 400, where the video frame containing the second video's lines is the next video frame after the frame containing the first video's lines. If the server 400 can find the second configuration file for the second video based on the first and second video's lines, where the second configuration file includes at least all audio data configured for the second video and the song information corresponding to the audio data, the server 400 will return the song information to the display device 200, where the song information is the song information for the audio data found in the second configuration file that matches the audio clip. If the server 400 still cannot find the second configuration file for the second video, the display device 200 will continue to send the third video's lines to the server 400 until the server 400 finds the corresponding second configuration file or the search times out.

[0333] For example, in the user interface shown in FIG35 , video player control B is playing a video and audio data simultaneously. The line "Life is like a box of chocolates" from the TV series currently playing is also displayed in the playback window of video player control B. After receiving a song identification command input by the user, display device 200 sends the line "Life is like a box of chocolates" from the TV series and the recorded audio clip to server 400. Server 400 first searches the audio database for songs 1, 2, and 3 based on the recorded audio clip. However, the line "Life is like a box of chocolates" cannot be used to identify a unique TV series; for example, TV series A, TV series B, and TV series C all contain this line. Therefore, display device 200 continues to send the line "You never know what you're going to get" contained in the next video frame to server 400.

[0334] Based on the lines "Life is like a box of chocolates" and "You never know what you're going to get," server 400 finds a unique TV series A. It then retrieves the configuration file corresponding to TV series A, which contains songs 3, 4, 5, etc. Based on the recorded audio clip, server 400 searches the audio database for songs 1, 2, and 3. These are matched against the configuration files for songs 3, 4, 5, etc., concluding that the source of the currently playing sound data is song 3. Server 400 then sends the song information for song 3 back to display device 200.

[0335] In some embodiments, if the active control is a music playback control that plays at least scrolling lyrics, the display device 200 sends the audio clip and the first lyric text of the scrolling lyrics to the server 400, wherein the audio clip is played while the scrolling lyrics are playing the first lyric text. The server 400 aligns the audio clip with the first lyric text and identifies the first song information of the sound data.

[0336] For example, in the user interface shown in FIG36 , a music player control is playing song M (which has been adapted into versions M1, M2, and M3, each with different lyrics. For example, version M1 includes lyrics AAAA, version M2 includes lyrics BBBB, and version M3 includes lyrics CCCC). The lyrics of song M, "Life is like a box of chocolates AAAA, you never know what you're going to get," are also displayed in the music player control's playback window. After receiving a song identification command input by the user, display device 200 sends the song lyrics and the recorded audio clip to server 400. Server 400 first searches an audio database for songs M1, M2, and M3 based on the recorded audio clip. Since songs M1, M2, and M3 have different lyrics, a unique song can be identified based on lyrics F. For example, if lyrics F match the lyrics of song M2, it can be determined that the song currently playing in the music player control is song M2. Finally, server 400 returns song information for song M2 to display device 200.

[0337] In some embodiments, if there is no active control in the current user interface, the condition for obtaining the second content data of the inactive control in the current page also includes that there is no active control running in the current foreground.

[0338] If there is no active control in the current user interface and there is an active control currently running in the foreground, after displaying the active control on the user interface, first content data of the active control is obtained, and the audio clip and the first content data are sent to the server 400. The server 400 identifies the first song information of the sound data based on the audio clip and the first content data, and receives the first song information fed back by the server.

[0339] For example, in the user interface shown in Figure 37, the current user interface displays a navigation bar, a sidebar, and multiple game introduction windows (containing only text and no dynamic content, therefore being inactive controls). A video player control is also running in the current foreground, but the video player control (the sound data also originates from the video player control) is dragged outside the current display window. Upon receiving a song identification instruction input by the user, the video player control can automatically move back to the current user interface. In the user interface shown in Figure 38, the video player control is displayed on the user interface. Then, according to the method of the above embodiment, the video name or lines and audio clip displayed on the video player control are sent to the server 400, and the server 400 searches for song information corresponding to the sound data based on the video name or lines and audio clip.

[0340] The specific implementation process of song information recognition in this embodiment can be shown in the signaling diagram shown in FIG39 , including the following steps:

[0341] S3901. The user enters a song identification instruction by clicking the song identification button in the video playback application page, and sends the song identification instruction to the song identification application (the user can also directly click the song identification application, in which case the video player still plays the current video normally), where the song identification application can be an audio playback application with a song identification function.

[0342] S3902. The song recognition application sends a recording system sound request to the audio output module, requesting an audio file (the audio output module may be an audio player, and the audio file is an audio clip of the sound data being played).

[0343] S3903. At the same time, the song identification application requests the application business system for the display text of the active control on the current interface, and the application business credit returns the display text of the active control on the current interface to the song identification application according to the request.

[0344] S3904. The audio output module returns the audio file to the song recognition application.

[0345] S3905. The application business system returns the display text of the activity control to the song identification application.

[0346] S3906. The song recognition application assembles data from the received audio file and the display text.

[0347] S3907. Then the song recognition application sends the assembled data to the song recognition cloud.

[0348] S3908: The song recognition cloud first searches for multiple matching audio data points based on the audio file, thereby simultaneously obtaining information about multiple songs. The song recognition cloud then further filters the obtained audio data points based on the displayed text to obtain unique song information corresponding to the audio data points. Finally, the song recognition cloud feeds the identified song information back to the song recognition application, which can then display the song information.

[0349] The present application provides a song information identification method. FIG40 is a flow chart of a song information identification method according to an exemplary embodiment. The song information identification method is applicable to the display device 200 in the system shown in FIG28. As shown in FIG40, the song information identification method may include the following steps:

[0350] In step S1001, the current user interface is displayed;

[0351] In step S1002, in response to the received song identification instruction, the sound data being played is collected to generate an audio clip and content data corresponding to the control in the current user interface is obtained, wherein the content data represents the media asset corresponding to the control;

[0352] In step S1003, the audio clip and the content data are sent to a server, so that the server identifies song information corresponding to the sound data based on the audio clip and the content data;

[0353] In step S1004, the identified song information fed back by the server is received and displayed.

[0354] The present application provides another song information identification method. FIG41 is a flow chart of a song information identification method according to an exemplary embodiment. The song information identification method is applicable to the display device 200 in the system shown in FIG28. As shown in FIG41, the song information identification method may include the following steps:

[0355] In step S2001, in response to the received song identification instruction, an audio segment of the sound data being played is collected.

[0356] In step S2002, if there is an active control in the current user interface, the first content data of the active control is obtained, and the audio clip and the first content data are sent to the server, so that the server identifies the first song information of the sound data based on the audio clip and the first content data, and receives the first song information fed back by the server, wherein the active control is a control with dynamic items.

[0357] In step S2003, if there is no active control in the current user interface, the second content data of the inactive control in the current page is obtained, and the audio clip and the second content data are sent to the server, so that the server identifies the second song information of the sound data based on the audio clip and the second content data, and receives the second song information fed back by the server, wherein the inactive control is a control without a dynamic item, the first content data and the second content data are different data, and the first song information and the second song information are different information.

[0358] In some embodiments, if the active control is a video playback control, a video name is displayed on the video playback control, and the sound data is the sound data being played by the video playback control, the audio clip and the video name are sent to the server, so that the server searches for a first configuration file corresponding to the first video with the video name, wherein the first configuration file includes at least all the audio data of the first video configuration and song information corresponding to the audio data; and receives the first song information fed back by the server, wherein the first song information is the song information of the audio data matching the audio clip found in the first configuration file.

[0359] In addition to the above embodiments regarding information identification, this application also provides embodiments regarding information recommendation, as described below:

[0360] As the types and number of songs continue to increase, when a user hears a song, he or she may not know the name of the song and hopes to learn related songs and related information of the song.

[0361] The music recognition function in a display device refers to the process of collecting the audio corresponding to the music playback clip and identifying the audio to obtain the corresponding song information. However, after the user uses the music recognition function, most display devices currently only display the information of the successfully identified songs, or other songs related to the successfully identified songs, ignoring the current playback scene of the user's display device, making the songs displayed by the display device inaccurate, resulting in a poor user experience.

[0362] It should be noted that, in the following embodiments, the processor 110 in the control device 100 is referred to as a third processor, the processor 250 in the display device 200 is referred to as a first processor, and the processor in the server is referred to as a second processor.

[0363] Figure 42 is a schematic diagram of the icon control interface display of an application in a display device according to one or more embodiments of the present application. As shown in Figure 42, the application layer includes at least one application that can display corresponding icon controls in the display, such as: live TV application icon controls, video on demand application icon controls, media center application icon controls, application center icon controls, game application icon controls, etc. The live TV application can provide live TV through different signal sources. The video on demand application can provide videos from different storage sources. Unlike the live TV application, the video on demand provides video display from certain storage sources. The media center application can provide applications for playing various multimedia content. The application center can provide storage for various applications.

[0364] In some embodiments, the display device is a terminal device with a display function, such as a television, a mobile phone, a personal computer, or a flat-screen TV.

[0365] In some embodiments of the present application, a display device is provided, comprising:

[0366] an output interface (display 260 and / or audio output interface 270 ), configured to output user interaction information;

[0367] The communicator 220 is used to communicate with the server 400;

[0368] Display 260.

[0369] In some embodiments, the first processor 250 is capable of receiving an operation input by a user to turn on the music recognition function, and in response to the operation, obtaining target information corresponding to the current playback scene of the display device, wherein the current playback scene includes a film and television playback scene or a song playback scene, and the target information under different current playback scenes is different. The target information is used to characterize the current playback scene and the songs played in the current playback scene, and then a recommendation request including the target information is sent to the server, so that the server obtains a target recommended song list based on the target information in the recommendation request, and the target recommended song lists corresponding to different target information are different. Finally, a recommendation response sent by the server is received, and the recommendation response includes: a target recommended song list, and the display 260 is controlled to display the target recommended song list.

[0370] In some embodiments of the present application, a server is provided, comprising:

[0371] The second processor is configured to: receive a recommendation request sent by the display device, the recommendation request including: target information, wherein the target information is obtained based on the current playback scene of the display device in response to the operation of turning on the music recognition function after the display device receives the user's input to turn on the music recognition function, the current playback scene includes a film and television playback scene or a song playback scene, and the target information in different current playback scenes is different. The target information is used to characterize the current playback scene and the songs played in the current playback scene, and then based on the target information, a target recommended song list is obtained, and a recommendation response is sent to the display device, the recommendation response including: the target recommended song list, so that the display device controls the display of the display device to display the target recommended song list after receiving the target recommended song list, wherein the target recommended song lists corresponding to different target information are different.

[0372] In some embodiments of the present application, the first processor of the display device receives an input operation to turn on the music recognition function, and in response to the operation to turn on the music recognition function, obtains target information corresponding to the current playback scene of the display device, wherein the current playback scene includes a film and television playback scene or a song playback scene, and the target information in different current playback scenes is different. The target information is used to characterize the current playback scene and the songs played in the current playback scene, and sends a recommendation request to the server, wherein the recommendation request includes: target information, so that the server obtains a target recommended song list based on the target information, and the target recommended song lists corresponding to different target information are different, receives a recommendation response sent by the server, and the recommendation response includes: target recommended song list, and controls the display of the display device to display the target recommended song list. In the above technical solution, combined with the current playback scene of the display device, the target information corresponding to the current playback scene is obtained, and the target recommended song list related to the target information is displayed through the display, which can make music recommendations more intelligent and accurate, thereby improving the user experience.

[0373] Figure 43 is a system framework diagram for performing music recommendation according to one or more embodiments of the present application. As shown in Figure 43, the system is deployed in a display device and may include an operation receiving module 401, an acquisition module 402, a sending module 403, and a display module 404.

[0374] The operation receiving module 401 in the system is used to receive an input operation to turn on the music recognition function. In some embodiments, the operation receiving module 401 can correspond to the user interface 280; the acquisition module 402 is used to respond to the operation of turning on the music recognition function and obtain the target information corresponding to the current playback scene of the display device, wherein the current playback scene includes a movie playback scene or a song playback scene, and the target information under different current playback scenes is different. The target information is used to characterize the current playback scene and the songs played in the current playback scene; the sending module 403 is used to send a recommendation request to the server, and the recommendation request includes: target information, so that the server obtains a target recommended song list based on the target information, and the target recommended song list corresponding to different target information is different; and the display module 404 is used to receive a recommendation response sent by the server, and the recommendation response includes: a target recommended song list, and controls the display to display the target recommended song list. In some embodiments, the acquisition module 402, the sending module 403, and the display module 404 can be functional units in an application executed by the first processor. Through the above process, combined with the current playback scene of the display device, the target information corresponding to the current playback scene is obtained, and the target recommended song list related to the target information is displayed through the display, which can make music recommendations more intelligent and accurate, thereby improving the user experience.

[0375] In order to explain the music recommendation method provided by the present application in more detail, it will be explained in an exemplary manner in combination with Figure 44 below. It can be understood that the steps involved in Figure 44 may include more steps or fewer steps in actual implementation, and the order of these steps may also be different, so as to be able to implement the music recommendation method provided in some embodiments of the present application. Some embodiments of the present application are not limited.

[0376] Figure 44 is a flowchart of a music recommendation method applied to a display device, provided in some embodiments of the present application. This embodiment can be applied to illustrate the music recommendation process on the display device side, and the process can be executed on the display device. As shown in Figure 44, the method specifically includes the following steps:

[0377] S501, receiving an input operation for starting a music recognition function.

[0378] In some embodiments, the operation of turning on the music recognition function can be triggered by the user by operating a button on the remote control, or by the user directly issuing a voice control command, or by other methods, which are not limited in this embodiment.

[0379] After the user triggers the operation of starting the music recognition function of the display device, the first processor of the display device can receive the operation input by the user to start the music recognition function.

[0380] S502: In response to the operation of starting the music recognition function, target information corresponding to the current playback scene of the display device is obtained.

[0381] In some embodiments, the current playback scene includes a movie playback scene or a song playback scene, and the target information in different current playback scenes is different.

[0382] In some embodiments, the target information is used to characterize the current playback scene and the song played in the current playback scene. In some embodiments, the target information may include a collected audio clip or a song identifier. The audio clip is agreed upon by the server and the display device through pre-set software settings to be used simultaneously to characterize the currently collected sound data for song recognition and to characterize the current first scene (film and television playback scene); the song identifier is agreed upon by the server and the display device through pre-set software settings to be used simultaneously to characterize the song corresponding to the currently collected sound data and to characterize the current second scene (song playback scene). In some embodiments, in some embodiments, the target information includes the sound clip and may also include the film and television identifier. The audio clip is agreed upon by the server and the display device to be used simultaneously to characterize the currently collected sound data for song recognition and to characterize the current first scene (film and television playback scene). The film and television identifier is used to filter recommendations related to the film and television identifier. The target information may include an audio clip and a media asset identifier, wherein the audio clip is agreed by the server and the display device to be used simultaneously to represent the currently collected sound data for song identification, and the media asset identifier is used to determine the current scene (for example, the film and television identifier is used to represent that the current scene is the first scene and / or is also used to filter recommendations related to the film and television identifier, and the song identifier is used to represent that the current scene is the second scene and / or is also used to filter recommendations related to the song identifier). In some embodiments, the film and television identifier is not used to represent that the current scene is the first scene but is used to filter recommendations related to the film and television identifier, and the song identifier is not used to represent that the current scene is the second scene but is used to filter recommendations related to the song identifier.

[0383] The video playback scene can be understood as the scene where the display device plays a video program. Video programs can include TV series, movies, variety shows, cartoons, etc., which are not limited in this embodiment. The song playback scene can be understood as the scene where the display device plays a song.

[0384] After the first processor of the display device receives the user input to turn on the music recognition function, it responds to the operation and obtains the program information corresponding to the display device, or the playback record, or other methods, etc., and can determine the current playback scene of the display device and the target information corresponding to the current playback scene. Through the target information, it can be determined whether the current playback scene is a film and television playback scene or a song playback scene, and can determine the song played in the current playback scene.

[0385] S503, sending a recommendation request to the server, wherein the recommendation request includes: target information, so that the server obtains a target recommended song list based on the target information, and different target information corresponds to different target recommended song lists.

[0386] Among them, the target recommended song list contains relevant information of at least one recommended song, and the relevant information may include information related to the recommended song, such as the name, singer, poster information, release time and composition information of the recommended song, etc. This embodiment does not make specific limitations on this.

[0387] The first processor of the display device sends a recommendation request to the server, the recommendation request including target information. The recommendation request is used to instruct the server to obtain a target recommended song list based on the target information in the recommendation request. After receiving the recommendation request sent by the display device, the server obtains song information corresponding to the target information based on the target information in the recommendation request, and then determines the target recommended song list based on the song information corresponding to the target information by calculating and querying song relevance.

[0388] It is understandable that when the target information is different, the target recommended song list determined by the server is also different.

[0389] S504: Receive a recommendation response sent by the server, the recommendation response including a target recommended song list, and control the display of the display device to display the target recommended song list.

[0390] After obtaining the target recommended song list, the server sends a recommendation response to the display device. Accordingly, the first processor of the display device can receive the recommendation response sent by the server, which includes the target recommended song list, and control the display of the display device to display the target recommended song list. The user can select songs of interest to play through the target recommended song list displayed on the display.

[0391] Some embodiments of the present application provide a music recommendation method applied to a display device. By combining the current playback scene of the display device, target information corresponding to the current playback scene is obtained, and a target recommended song list related to the target information is displayed on the display. This can make music recommendations more intelligent and accurate, thereby improving the user experience.

[0392] For example, FIG45 is a schematic diagram of an interaction process between a display device and a server provided in some embodiments of the present application. As shown in FIG45 , the interaction process between the display device and the server specifically includes the following steps:

[0393] S5001: The first processor receives an input operation for starting a music recognition function. Please refer to the explanation in S501 above and will not be repeated here.

[0394] S5002: In response to the operation of starting the music recognition function, the first processor obtains target information corresponding to the current playback scene of the display device.

[0395] Please refer to the explanation in S502 above, which will not be repeated here.

[0396] S5003: The first processor sends a recommendation request to the server, where the recommendation request includes target information.

[0397] Please refer to the explanation in S503 above, which will not be repeated here.

[0398] S5004: The second processor receives a recommendation request sent by the display device, where the recommendation request includes target information.

[0399] After the first processor of the display device sends a recommendation request to the server, the second processor of the server can correspondingly receive the recommendation request sent by the display device.

[0400] S5005: The second processor obtains a target recommended song list based on the target information.

[0401] After the second processor receives the recommendation request sent by the display device, it obtains song information corresponding to the target information based on the target information in the recommendation request, and then determines the target recommended song list through song relevance calculation and query based on the song information corresponding to the target information.

[0402] S5006: The second processor sends a recommendation response to the display device, where the recommendation response includes a target recommended song list.

[0403] After obtaining the target recommended song list, the second processor sends a recommendation response to the display device.

[0404] S5007, the first processor receives a recommendation response sent by the server, the recommendation response including: a target recommended song list, and controls the display to display the target recommended song list.

[0405] Please refer to the explanation in S504 above, which will not be repeated here.

[0406] In some embodiments, optionally, the method further comprises:

[0407] When the target song corresponding to the target information is stored in the target recommended song list, the target song has a separate label, and the label is used to indicate that the target song is a song played in the current playback scenario.

[0408] In some embodiments, the target song is stored in a target recommended song list, and then the target recommended song list is sent to a display device for subsequent display presentation.

[0409] In some embodiments, optionally, the method further comprises:

[0410] When the target song corresponding to the target information is not stored in the target recommended song list, the target song corresponding to the target information is used as a recognition result, and the recognition result and the target recommended song list are sent to a display device.

[0411] The recognition result may include the target song, or may not match the target song.

[0412] In some embodiments, by using the target song as the recognition result, the recognition result and the target recommended song list are sent together to the display device for subsequent display presentation.

[0413] It should be noted that: by the above two methods of storing the target song in the target recommended song list and not storing the target song in the target recommended song list, the server can obtain the target recommended song list in a more flexible manner.

[0414] In some embodiments, optionally, the display includes a first area and a second area when displaying the target recommended song list;

[0415] In a case where the target song corresponding to the target information is stored in the target recommended song list, controlling the display to display the target recommended song list includes:

[0416] When a target song corresponding to the target information exists in the target recommended song list, controlling the first area to display information corresponding to the target song, and controlling the second area to display information corresponding to the remaining recommended songs in the target recommended song list except the target song;

[0417] or,

[0418] When there is no target song corresponding to the target information in the target recommended song list, the first area is controlled to display prompt information that the target song is not matched, and the second area is controlled to display information corresponding to the recommended songs in the target recommended song list.

[0419] In some embodiments, when the target song corresponding to the target information is stored in the target recommended song list, when the target song corresponding to the target information exists in the target recommended song list, or the target song corresponding to the target information does not exist in the target recommended song list, the target recommended song list is displayed through the above-mentioned different display methods, so that the display page is clearer and easier for users to view.

[0420] For example, Figure 46 is a schematic diagram of a first display interface of a display provided by some embodiments of the present application when displaying a target recommended song list. As shown in Figure 46, the first area displays the target song corresponding to the target information in the target recommended song list, and the second area displays the information corresponding to the remaining recommended songs in the target recommended song list except the target song.

[0421] For example, Figure 47 is a schematic diagram of a second display interface of a display provided by some embodiments of the present application when displaying a target recommended song list. As shown in Figure 47, the first area displays a prompt message indicating that no target song has been matched, and the second area displays information corresponding to the recommended songs in the target recommended song list.

[0422] It should be noted that the number of songs displayed in the second area in Figures 46 and 47 is only for illustrative purposes and is not intended to limit it.

[0423] In some embodiments, optionally, obtaining target information corresponding to the current playback scene of the display device includes:

[0424] When the current playback scene is a film and television playback scene, based on the film and television playback scene, obtaining audio clip information and video identification information corresponding to the film and television playback scene, wherein the video identification information is identification information corresponding to the target video played in the film and television playback scene;

[0425] The sending of a recommendation request to the server, wherein the recommendation request includes the target information, so that the server obtains a target recommended song list based on the target information, includes:

[0426] The recommendation request is sent to the server, wherein the recommendation request includes: the audio segment information and the video identification information, so that the server obtains the target recommended song list according to the audio segment information and the video identification information.

[0427] The video identification information may be media asset identification information, such as the serial number of the media asset content played in a film or television playback scenario, or other information, which is not limited in this embodiment. The media asset identification information can be understood as the identification information of the media asset content (i.e., the target video) played in the film or television playback scenario. The preset duration can be predetermined or determined based on the specific situation, which is not limited in this embodiment.

[0428] In some embodiments, when the current playback scene is a film or TV scene, the first processor of the display device can obtain audio segment information corresponding to the film or TV scene based on the film or TV scene. For example, the first processor can extract a preset duration from the sound data played in the film or TV scene as the audio segment information, or can record the sound data played in the film or TV scene to obtain the audio segment information. The audio segment information can also be obtained through other methods, which are not limited in this embodiment. After obtaining the audio segment information, the first processor sends a recommendation request to the server, the recommendation request including the audio segment information and video identification information. The recommendation request is used to instruct the server to obtain a target recommended song list based on the audio segment information and video identification information. After the server receives the audio clip information and the video identification information, it determines a first recommended song list formed by film and television songs associated with the video identification information based on the video identification information through a corresponding recommendation algorithm, and determines the target audio identification information corresponding to the audio clip information based on the audio clip information through a corresponding recognition algorithm, and determines a second recommended song list associated with the target audio identification information based on the target audio identification information, and performs a deduplication operation on the first recommended song list and the second recommended song list, that is: for the recommended songs with the same song information in the first recommended song list and the second recommended song list, one recommended song is retained and the remaining duplicate recommended songs are deleted, and then the deduplicated recommended songs are sorted, and the relevant information of each recommended song is spliced ​​together to obtain the target recommended song list. Among them, the sorting can be based on the popularity of the song, the release time of the song, or other information, etc., which are not specifically limited here.

[0429] In some embodiments, by sending a recommendation request including audio clip information and video identification information to the server in a film and television playback scenario, it is helpful to make the target recommended song list obtained by the server more accurate and more in line with user needs.

[0430] In some embodiments, the display device is product A, and the currently playing program is a song from film and television program A: XX Legend (movie title). Display device A obtains the corresponding audio clip information 1 and video identification information 1 (i.e., the media asset identification information of XX Legend) in the film and television playback scene, and sends a recommendation request containing the audio clip information 1 and video identification information 1 to the server. Accordingly, after receiving the recommendation request sent by display device A, the server can determine, based on the video identification information 1 in the recommendation request, that the display device is currently playing XX Legend. A first recommended song list can be obtained based on the film and television songs associated with XX Legend, and based on the audio clip information 1, the target audio identification information corresponding to the audio clip information 1 (i.e., the identification information of the song corresponding to the audio clip information 1) can be determined. Then, the name of the target audio is determined, and a correlation calculation is performed based on the name of the target audio to obtain a second recommended song list. The first recommended song list and the second recommended song list are deduplicated, the deduplicated recommended songs are sorted, and the relevant information of each recommended song is spliced ​​to obtain a target recommended song list 1 so that the song recommendation list prioritizes songs related to the film and television.

[0431] In some embodiments, after determining a list of songs to be recommended based on the audio clip information 1, the server may filter and sort songs related to the film and television identifiers from the list of songs to be recommended, so that songs related to the film and television are displayed first in the generated target recommended song list.

[0432] In some embodiments, the display device is product B (product B is different from product A, and may be different from product A in manufacturer, brand, or version, etc., which is not limited in this embodiment), and the currently playing song is still the same song in film and television program A: XX Legend (the same as the film and television program and song currently played by display device A in the first case), and display device B obtains the corresponding audio clip information 2 and video identification information 2 (i.e., the media asset identification information of XX Legend) in the film and television playback scene, and sends a recommendation request containing audio clip information 2 and video identification information 2 to the server. Accordingly, after receiving the recommendation request sent by display device B, the server can obtain the target recommended song list 2 according to the video identification information 2 and audio clip information 2 in the recommendation request, wherein the method for determining the target recommended song list 2 is the same as the method for determining the target recommended song list 1, but because product B is different from product A, the target recommended song list 1 is also different from the target recommended song list 2.

[0433] In some embodiments, when the film and television programs and songs played in the same film and television playback scene are the same, the target recommended song lists determined are different when the display devices are different.

[0434] In some embodiments, the display device is product A, and the currently playing song is from film program B: XX Hai (movie title). Display device A obtains the audio segment information 3 and video identification information 3 (i.e., the media asset identification information of XX Hai) corresponding to the film and television playback scene, and sends a recommendation request containing the audio segment information 3 and the video identification information 3 to the server. Accordingly, after receiving the recommendation request sent by display device A, the server can obtain a target recommended song list 3 based on the video identification information 3 and the audio segment information 3 in the recommendation request. The method for determining target recommended song list 3 is the same as the method for determining target recommended song lists 1 and 2.

[0435] In some embodiments, the display device is product A, and the song currently playing is from film and television program C: XX era (movie name) (the song is the same as the song in the third case, but the corresponding film and television program is different). Display device A obtains the audio clip information 4 and video identification information 4 corresponding to the film and television playback scene (i.e., the media asset identification information of XX era), and sends a recommendation request containing audio clip information 4 and video identification information 4 to the server. Accordingly, after receiving the recommendation request sent by display device A, the server can obtain the target recommended song list 4 based on the video identification information 4 and audio clip information 4 in the recommendation request. The method for determining the target recommended song list 4 is the same as the method for determining the target recommended song lists 1, 2, and 3. However, since film and television program B and film and television program C are different, the target recommended song list 3 is also different from the target recommended song list 4.

[0436] In some embodiments, when different film and television programs are played in a film and television playback scenario, but the sound information of the played songs is the same, the determined target recommended song lists are different.

[0437] In some embodiments, after receiving the sound clip and the media asset identifier, the server determines whether the media asset identifier is a film or television identifier or a song identifier. If it is a film or television identifier, after determining the songs to be recommended based on the sound clip, it is necessary to further filter according to the film or television identifier to generate a recommendation list and feedback. If it is a song identifier, after determining the songs to be recommended based on the sound clip, it is not necessary to further filter according to the song identifier to directly generate a recommendation list and feedback.

[0438] In some embodiments, the server determines, based on the target audio identification information, a second recommended song list associated with the target audio identification information, including:

[0439] The server determines the reference information corresponding to the target audio based on the target audio identification information. The reference information includes at least one of the song name, singer name, song theme and historical playback data corresponding to the target audio identification information. Based on the target information, the second recommended song list can be determined through song relevance calculation and query.

[0440] In some embodiments, determining the second recommendation list by the above method is simple, efficient, and fast.

[0441] In some embodiments, optionally, when the current playback scene is a movie playback scene, sending the recommendation request to the server includes:

[0442] Sending a recognition request to the server, wherein the recognition request includes: the audio segment information;

[0443] receiving a recognition response sent by the server, wherein the recognition response includes: target audio identification information corresponding to the audio segment information;

[0444] A song recommendation request is sent to the server, wherein the song recommendation request includes: video identification information and the target audio identification information, so that the server obtains the target recommended song list according to the video identification information and the target audio identification information.

[0445] The song recommendation request may be understood as a sub-request included in the recommendation request.

[0446] In some embodiments, when the current playback scene is a film and television playback scene, the first processor of the display device sends an identification request to the server, and the identification request includes: audio clip information. After the server receives the identification request, it identifies the audio clip information included in the identification request through the identification algorithm. After obtaining the identification result, it sends an identification response to the display device, and the identification response includes: target audio identification information corresponding to the audio clip information (i.e., the identification result). Accordingly, the first processor can receive the identification response sent by the server, and then send a song recommendation request to the server, and the song recommendation request includes: video identification information and target audio identification information. After the server receives the song recommendation request, it can obtain a target recommended song list based on the video identification information and target audio identification information included in the song recommendation request.

[0447] In some embodiments, the first processor of the display device first sends an identification request to the server and then sends a song recommendation request to the server, so that the first processor can promptly obtain the target audio identification information corresponding to the audio segment information included in the identification response, and facilitates the subsequent server to obtain the target recommended song list.

[0448] In some embodiments, optionally, when the current playback scene is a movie playback scene, sending the recommendation request to the server includes:

[0449] Sending a recognition request to the server, wherein the recognition request includes: the audio segment information;

[0450] receiving a recognition response sent by the server, wherein the recognition response includes: target audio identification information corresponding to the audio segment information not being recognized;

[0451] A song recommendation request is sent to the server, wherein the song recommendation request includes video identification information, so that the server obtains the target recommended song list according to the video identification information.

[0452] In some embodiments, when the current playback scene is a film or TV playback scene, the first processor of the display device sends an identification request to the server, the identification request including audio segment information. After receiving the identification request, the server performs identification based on the audio segment information included in the identification request. If no identification result is obtained, the server sends an identification response to the display device, the identification response including target audio identification information corresponding to the unidentified audio segment information. Accordingly, the first processor can receive the identification response sent by the server and then send a song recommendation request to the server, the song recommendation request including video identification information. After receiving the song recommendation request, the server can obtain a target recommended song list based on the video identification information included in the song recommendation request.

[0453] In some embodiments, the first processor of the display device first sends an identification request to the server and then sends a song recommendation request to the server, so that the first processor can promptly obtain the target audio identification information corresponding to the unrecognized audio segment information, and facilitates the subsequent server to obtain the target recommended song list based on the video identification information.

[0454] In some embodiments, optionally, obtaining target information corresponding to the current playback scene of the display device includes:

[0455] When the current playback scene is a song playback scene, based on the song playback scene, obtaining audio identification information corresponding to the song playback scene;

[0456] The sending of a recommendation request to the server, wherein the recommendation request includes the target information, so that the server obtains a target recommended song list based on the target information, includes:

[0457] The recommendation request is sent to the server, where the recommendation request includes the audio identification information, so that the server obtains the target recommended song list based on the audio identification information.

[0458] The song playback scene may include: a scene in which a song player plays music, or a karaoke scene corresponding to a karaoke player, which is not limited in this embodiment. In the scene in which the song player plays music, the audio identification information may be the identification information of the currently playing song. In the karaoke scene, the audio identification information may be the accompaniment identification information of the current song or the original sound identification information of the current song, etc., depending on the actual situation, which is not limited in this embodiment.

[0459] It can be understood that when the current playing scene is a song playing scene, the target information is: audio identification information.

[0460] In some embodiments, when the current playback scene is a song playback scene, the first processor of the display device can directly obtain the audio identification information of the corresponding song in the song playback scene based on the song playback scene by obtaining program information or playback records. After obtaining the audio identification information, the first processor sends a recommendation request to the server, and the recommendation request includes: audio identification information, and the recommendation request is used to instruct the server to obtain a target recommended song list based on the audio identification information. After receiving the audio identification information, the server can determine the songs associated with the audio identification information based on the audio identification information, and specifically can sort the associated songs and splice the relevant information of each associated song to obtain a target recommended song list.

[0461] In some embodiments, since the corresponding audio identification information in the song scenario does not need to be identified and the display device can obtain it directly, in the song playback scenario, by sending a recommendation request including audio identification information to the server, it is beneficial to improve the efficiency of the server in obtaining the target recommended songs and save time.

[0462] In some embodiments, the server determines, based on the audio identification information, a song associated with the audio identification information, including:

[0463] The server determines the corresponding reference information based on the audio identification information. The reference information includes at least one of the song name, singer name, song theme and historical playback data corresponding to the audio identification information. According to the target information, the song related to the audio identification information can be determined through song relevance calculation and query.

[0464] For example, in a karaoke scene, the first processor of the display device obtains the audio identification information corresponding to the karaoke scene, such as the identification information of XX accompaniment, and then sends a recommendation request to the server, which includes: the identification information of XX accompaniment, so that the server obtains the corresponding target recommended song list based on the identification information of XX accompaniment.

[0465] In the above embodiment, the current playback scene is determined by the display device and the target information corresponding to the current playback scene of the display device is obtained. Specifically, when the current playback scene is a film and television playback scene, the target information is: audio clip information and video identification information; when the current playback scene is a song playback scene, the target information is audio identification information. In the above two cases, after receiving the recommendation request, the server determines the target recommended song list based on the target information included in the recommendation request. If the target information includes audio clip information, the target recommended song list is determined based on the audio clip information and video identification information; if the target information does not include audio clip information, the target recommended song list is determined based on the video identification information. Therefore, the server does not need to determine whether the current playback scene is a film and television playback scene or a song playback scene, which facilitates the subsequent server to obtain the target recommended song list and send a recommendation response based on the target information. The server adopts a corresponding method to determine the target recommended song list based on the different target information, making the method of determining the target recommended song list more flexible and accurate, and avoiding errors.

[0466] In some embodiments, optionally, obtaining target information corresponding to the current playback scene of the display device may further include:

[0467] When the current playback scene is a film and television playback scene, based on the film and television playback scene, obtaining audio clip information, video identification information, and first channel identification information corresponding to the film and television playback scene, wherein the first channel identification information is the channel identification information corresponding to the film and television playback scene;

[0468] The sending of a recommendation request to the server, wherein the recommendation request includes the target information, so that the server obtains a target recommended song list based on the target information, may further include:

[0469] The recommendation request is sent to the server, wherein the recommendation request includes: the audio clip information, the video identification information and the first channel identification information, so that the server obtains the target recommended song list according to the audio clip information, the video identification information and the first channel identification information.

[0470] In some embodiments, when the current playback scene is a film and television playback scene, the target information may be: audio clip information, video identification information, and first channel identification information. The display device can obtain the audio clip information, video identification information, and first channel identification information based on the target video being played in the film and television playback scene, and send a recommendation request containing the audio clip information, video identification information, and first channel identification information to the server. Accordingly, after receiving the recommendation request containing the audio clip information, video identification information, and first channel identification information, the server can determine that the current playback scene is a film and television playback scene based on the first channel identification information, and determine a target recommended song list based on the audio clip information, video identification information, and first channel identification information using a corresponding recommendation algorithm.

[0471] In some embodiments, in a film and television playback scenario, the target information is: based on the audio clip information and video identification information, the first channel identification information is added, and the audio clip information, video identification information and the first channel identification information are used as the target information, which is beneficial for the server to distinguish the current playback scene and determine the target recommended song list.

[0472] In some embodiments, optionally, obtaining target information corresponding to the current playback scene of the display device may further include:

[0473] When the current playback scene is a song playback scene, based on the song playback scene, obtaining audio identification information and second channel identification information corresponding to the song playback scene, wherein the second channel identification information is the channel identification information corresponding to the song playback scene;

[0474] The sending of a recommendation request to the server, wherein the recommendation request includes the target information, so that the server obtains a target recommended song list based on the target information, may further include:

[0475] The recommendation request is sent to the server, wherein the recommendation request includes: the audio identification information and the second channel identification information, so that the server obtains the target recommended song list according to the audio identification information and the second channel identification information.

[0476] In some embodiments, when the current playback scene is a song playback scene, the target information may be: audio identification information and second channel identification information. The display device can obtain the audio identification information and second channel identification information based on the songs played in the song playback scene, and send a recommendation request containing the audio identification information and the second channel identification information to the server. Accordingly, after receiving the recommendation request containing the audio identification information and the second channel identification information, the server can determine that the current playback scene is a song playback scene based on the second channel identification information, and determine a target recommended song list based on the audio identification information and the second channel identification information using a corresponding recommendation algorithm.

[0477] In some embodiments, in a song playback scenario, the target information is: based on the audio identification information, the second channel identification information is added, and the audio identification information and the second channel identification information are used as the target information, which is beneficial for the server to distinguish the current playback scenario and determine the target recommended song list.

[0478] In some embodiments, optionally, obtaining target information corresponding to the current playback scene of the display device may specifically include:

[0479] When the current playback scene is a film and television playback scene, based on the film and television playback scene, obtaining first target audio segment information and first channel identification information corresponding to the film and television playback scene, wherein the first channel identification information is the channel identification information corresponding to the film and television playback scene;

[0480] Sending a recommendation request to a server, wherein the recommendation request includes: the target information, so that the server obtains a target recommended song list based on the target information, which may specifically include:

[0481] Sending the recommendation request to the server, wherein the recommendation request includes: the first target audio segment information and the first channel identification information, so that the server obtains the target recommended song list according to the first target audio segment information and the first channel identification information;

[0482] or,

[0483] The acquiring of target information corresponding to the current playback scene of the display device may specifically include:

[0484] When the current playback scene is a song playback scene, based on the song playback scene, obtaining second target audio segment information and second channel identification information corresponding to the song playback scene, wherein the second channel identification information is the channel identification information corresponding to the song playback scene;

[0485] Sending a recommendation request to a server, wherein the recommendation request includes: the target information, so that the server obtains a target recommended song list based on the target information, which may specifically include:

[0486] The recommendation request is sent to the server, where the recommendation request includes: the second target audio segment information and the second channel identification information, so that the server obtains the target recommended song list according to the second target audio segment information and the second channel identification information.

[0487] The first target audio segment information is obtained in the same manner as the audio segment information, and is not limited here. The second target audio segment information can be captured from song data played in the song playback scenario, or recorded from the song data played in the song playback scenario. The second target audio segment information can also be obtained by other means, which are not limited in this embodiment.

[0488] In some embodiments, when the current playback scene is a film and television playback scene, the target information may be: first target audio segment information and first channel identification information. The display device can obtain the first target audio segment information and first channel identification information based on the target video played in the film and television playback scene (the display device only obtains the target information and does not determine the current scene of the display device), and sends a recommendation request containing the first target audio segment information and the first channel identification information to the server. Accordingly, after receiving the recommendation request containing the first target audio segment information and the first channel identification information, the server can determine that the current playback scene is a film and television playback scene based on the first channel identification information, and determine the target recommended song list based on the first target audio segment information and the first channel identification information through the corresponding recommendation algorithm; when the current playback scene is a song playback scene, the target information may be: second target audio segment information and second channel identification information. The display device can obtain the second target audio segment information and the second channel identification information according to the song played in the song playback scene (the display device only obtains the target information and does not determine the current scene of the display device), and sends a recommendation request containing the second target audio segment information and the second channel identification information to the server. Accordingly, after receiving the recommendation request containing the second target audio segment information and the second channel identification information, the server can determine that the current playback scene is a song playback scene according to the second channel identification information, and determine the target recommended song list through the corresponding recommendation algorithm based on the second target audio segment information and the second channel identification information.

[0489] In some embodiments, in a film and television playback scenario, the target information may further include first target audio clip information and first channel identification information. In a song playback scenario, the target information may further include second target audio clip information and second channel identification information. In both of these cases, the processor of the display device does not need to determine the current playback scenario. After receiving the recommendation request, the server determines the current playback scenario based on the specific information included in the recommendation request. In some embodiments, if the recommendation request includes first target audio clip information and first channel identification information, the server determines the current playback scenario as a film and television playback scenario based on the first channel identification information and obtains a target recommended song list based on the first target audio clip information and the first channel identification information. If the recommendation request includes second target audio clip information and second channel identification information, the server determines the current playback scenario as a song playback scenario based on the second channel identification information and obtains a target recommended song list based on the second target audio clip information and the second channel identification information. By receiving the recommendation request containing the target information, the server determines the current playback scenario based on the target information and adopts a corresponding method to determine the target recommended song list, making the target recommended song list more accurate and more in line with user needs, thereby improving the user experience.

[0490] In some embodiments, after the user inputs the operation to turn on the music identification function and before the display device controls the display to display the target recommended song list, the display device controls the display to display the same music identification interface, which may include the following prompt information: Music identification is in progress, please wait.

[0491] In some embodiments, the above process can ensure that the user is unaware when the current playback scene is different.

[0492] In some embodiments, optionally, the method further comprises:

[0493] In response to an operation of playing a target recommended song in the target recommended song list, controlling the display device to switch from an interface corresponding to the target recommended song list to a playback interface of a player corresponding to the target recommended song, wherein the target recommended song is any one of the recommended songs in the target recommended song list;

[0494] In response to a return operation triggered by the user, the display device is controlled to switch from the playback interface to the interface corresponding to the target recommended song list.

[0495] In some embodiments, the first processor of the display device can control the display device to switch from the interface corresponding to the target recommended song list to the playback interface of the player corresponding to the target recommended song in response to the operation of playing the target recommended song in the target recommended song list, and control the display device to switch from the playback interface to the interface corresponding to the target recommended song list in response to the return operation triggered by the user, which can facilitate the user's operation.

[0496] For example, when the current playback scene is a film and television playback scene, when the user clicks on the target recommended song in the target recommended song list (the target recommended song list can be displayed in the corresponding playback interface under the film and television playback scene) to play it, the first processor of the display device, in response to the operation of playing the target recommended song, controls the display device to switch from the interface corresponding to the target recommended song list to the playback interface of the player corresponding to the target recommended song, and the play list in the playback interface is the target recommended song list; if the user clicks the return button, the first processor controls the display device to switch from the playback interface to the interface corresponding to the target recommended song list in response to the return operation triggered by the user. At this time, the display device can resume playback from the corresponding playback time point in the film and television playback scene before the switch, avoiding manual operation by the user.

[0497] For example, when the current playback scene is a scene in which a song player plays music, the display device runs the song player and records the page address of the last non-song player. When the user clicks on the target recommended song list (the target recommended song list can be displayed in the playback interface corresponding to the song player) to play the target recommended song, the first processor of the display device, in response to the operation of playing the target recommended song, controls the display device to switch from the interface corresponding to the target recommended song list to the playback interface of the player corresponding to the target recommended song (the player at this time is still the song player before switching), and the play list in the song player is updated to the target recommended song list; if the user clicks the return key, the first processor controls the display device to switch from the playback interface to the interface corresponding to the target recommended song list in response to the return operation triggered by the user. If the user clicks the return key again, the first processor controls the display device to return to the interface corresponding to the page address of the last non-song player.

[0498] Exemplarily, when the current playback scene is a karaoke scene, the display device runs a karaoke player and records the page address of the last non-karaoke player. When the user clicks on the target recommended song list (the target recommended song list can be displayed in the playback interface corresponding to the song player) to play, if the target recommended song is a non-accompaniment song, the first processor of the display device, in response to the operation of playing the target recommended song, controls the display device to switch from the interface corresponding to the target recommended song list to the playback interface of the player corresponding to the target recommended song (i.e., from the interface corresponding to the karaoke player to the interface corresponding to the song player). In this case, if the user clicks the return button, the first processor controls the display device to switch from the playback interface to the interface corresponding to the target recommended song list in response to the return operation triggered by the user, i.e., switch to the karaoke player interface. corresponding interface; if the target recommended song is an accompaniment song, the first processor of the display device, in response to the operation of playing the target recommended song, controls the display device to switch from the interface corresponding to the target recommended song list to the playback interface of the player corresponding to the target recommended song (that is, the player corresponding to the target recommended song is still the karaoke player). In this case, if the user clicks the return button, the first processor controls the display device to switch from the playback interface to the interface corresponding to the target recommended song list in response to the return operation triggered by the user, that is, to switch to the last corresponding interface of the karaoke player. If the user clicks the return button again, the first processor controls the display device to return to the interface corresponding to the last page address that is not a karaoke player.

[0499] In some embodiments, when the current playback scene is different, the switched interface and the returned interface may be different. Through the above process, the user experience can be improved and the user's manual operations can be reduced.

[0500] For example, Figure 48 is a schematic diagram of the interaction process between a display device and a server in a film and television playback scenario provided by some embodiments of the present application. This embodiment mainly describes the process of the server identifying target audio identification information in a film and television playback scenario and implementing music recommendations through the interaction between the display device and the server. As shown in Figure 48, the interaction process specifically includes the following steps:

[0501] S6001: The first processor receives an input operation for starting a music recognition function.

[0502] Please refer to the explanation in S501 above, which will not be repeated here.

[0503] S6002: When the current playback scene is a film and television playback scene, the first processor obtains audio segment information and video identification information corresponding to the film and television playback scene in response to an operation of enabling a music recognition function.

[0504] S6003: The first processor sends a recognition request to the music recognition service, where the recognition request includes audio segment information.

[0505] The music recognition service can be understood as a module in the server for identifying audio clip information.

[0506] After obtaining the audio segment information and video identification information corresponding to the film and television playback scene, the first processor sends a recognition request containing the audio segment information to the music recognition service.

[0507] S6004: The music recognition service receives the recognition request sent by the display device and determines the corresponding target audio identification information based on the audio segment information.

[0508] After receiving the recognition request sent by the display device, the music recognition service recognizes the audio segment information included in the recognition request and can determine the corresponding target audio identification information.

[0509] S6005: The music recognition service sends a recognition response to the display device, where the recognition response includes target audio identification information.

[0510] After the music recognition service determines the target audio identification information, it sends a recognition response to the display device, where the recognition response includes the target audio identification information.

[0511] S6006: The first processor sends a song recommendation request to the recommendation service, where the song recommendation request includes video identification information and target audio identification information.

[0512] Among them, the recommendation service can be understood as a module in the server for making song recommendations.

[0513] After receiving the identification response sent by the recommendation service, the first processor sends a song recommendation request to the recommendation service.

[0514] S6007: Receive a song recommendation request sent by the display device, and obtain a target recommended song list based on the video identification information and the target audio identification information.

[0515] After receiving the song recommendation request sent by the display device, the recommendation service can obtain a target recommended song list based on the video identification information and the target audio identification information.

[0516] S6008, the recommendation service sends a song recommendation response to the display device, where the song recommendation response includes: a target recommended song list.

[0517] After obtaining the target recommended song list, the recommendation service sends a song recommendation response to the display device.

[0518] S6009, the first processor receives a song recommendation response sent by the recommendation service, the song recommendation response including: a target recommended song list, and controls the display to display the target recommended song list.

[0519] For example, Figure 49 is a schematic diagram of the interaction process between a display device and a server in another film and television playback scenario provided by some embodiments of the present application. This embodiment mainly describes the process of implementing music recommendations through interaction between a display device and a server when the server fails to recognize the target audio identification information in a film and television playback scenario. As shown in Figure 49, the interaction process specifically includes the following steps:

[0520] S7001: The first processor receives an input operation for starting a music recognition function.

[0521] S7002: When the current playback scene is a film and television playback scene, the first processor obtains audio segment information and video identification information corresponding to the film and television playback scene in response to an operation of enabling a music recognition function.

[0522] S7003: The first processor sends a recognition request to the music recognition service, where the recognition request includes audio segment information.

[0523] S7004: The music recognition service receives the recognition request sent by the display device and determines the corresponding target audio identification information based on the audio segment information.

[0524] Among them, S7001-S7004 refer to the explanation of S6001-S6004 and are not repeated here.

[0525] S7005: The music recognition service sends a recognition response to the display device, where the recognition response includes: target audio identification information corresponding to the unrecognized audio segment information.

[0526] When the music recognition service fails to recognize the target audio identification information corresponding to the audio segment information, the music recognition service sends a recognition response to the display device, where the recognition response includes: the target audio identification information corresponding to the audio segment information is not recognized.

[0527] S7006: The first processor sends a song recommendation request to the recommendation service, where the song recommendation request includes video identification information.

[0528] The first processor receives an identification response sent by the music identification service. Since the identification response includes: target audio identification information corresponding to the audio segment information is not identified, the first processor sends a song recommendation request to the recommendation service, and the song recommendation request includes: video identification information.

[0529] S7007, the recommendation service receives the song recommendation request sent by the display device, and obtains a target recommended song list based on the video identification information.

[0530] After receiving the song recommendation request sent by the display device, the recommendation service obtains a target recommended song list according to the video identification information in the song recommendation request.

[0531] S7008, the recommendation service sends a song recommendation response to the display device, where the song recommendation response includes: a target recommended song list.

[0532] S7009, the first processor receives a song recommendation response sent by the recommendation service, the song recommendation response including: a target recommended song list, and controls the display to display the target recommended song list.

[0533] For example, FIG50 is a schematic diagram of the interaction process between a display device and a server in a song playback scenario provided by some embodiments of the present application. This embodiment mainly illustrates the process of implementing music recommendations through interaction between a display device and a server in a song playback scenario. As shown in FIG50 , the interaction process specifically includes the following steps:

[0534] S8001, receiving an input operation for starting a music recognition function.

[0535] Please refer to the explanation in S501 above, which will not be repeated here.

[0536] S8002, when the current playing scene is a song playing scene, in response to an operation of turning on a music recognition function, obtaining audio identification information corresponding to the song playing scene.

[0537] S8003: The first processor sends a recommendation request to the recommendation service, where the recommendation request includes audio identification information.

[0538] After acquiring the audio identification information, the first processor sends a recommendation request to the recommendation service.

[0539] S8004: The recommendation service receives the recommendation request sent by the display device and obtains a target recommended song list based on the audio identification information.

[0540] After receiving the recommendation request sent by the display device, the recommendation service can obtain a target recommended song list according to the audio identification information.

[0541] S8005, the recommendation service sends a recommendation response to the display device, where the recommendation response includes: a target recommended song list.

[0542] After the recommendation service obtains the target recommended song list, it sends a recommendation response to the display device.

[0543] S8006, the first processor receives a recommendation response sent by the recommendation service, the recommendation response including: a target recommended song list, and controls the display to display the target recommended song list.

[0544] For example, FIG51 is a flowchart of a music recommendation method applied to a server according to some embodiments of the present application. This embodiment can be applied to illustrate the music recommendation process on the server side. As shown in FIG51 , the method specifically includes the following steps:

[0545] S910: Receive a recommendation request sent by a display device, where the recommendation request includes target information.

[0546] Among them, the target information is obtained based on the current playback scene of the display device in response to the operation of turning on the music recognition function after the display device receives the input, wherein the current playback scene includes a film and television playback scene or a song playback scene. The target information in different current playback scenes is different, and the target information is used to represent the current playback scene and the songs played in the current playback scene.

[0547] When a user triggers the operation of turning on the music recognition function of the display device, the first processor of the display device can receive the operation and, in response to the operation, obtain target information corresponding to the current playback scene. At the same time, the first processor sends a recommendation request containing the target information to the server. In response, the second processor of the server can receive the recommendation request sent by the display device.

[0548] S920, based on the target information, obtain a target recommended song list, and send a recommendation response to the display device, wherein the recommendation response includes: the target recommended song list, so that the display device controls the display of the display device to display the target recommended song list.

[0549] Among them, the target recommended song lists corresponding to different target information are different.

[0550] After the second processor of the server receives the recommendation request sent by the display device, it obtains the song information corresponding to the target information based on the target information in the recommendation request, and then determines the target recommended song list through song relevance calculation and query based on the song information corresponding to the target information.

[0551] In some embodiments, a music recommendation method applied to a server is provided, in which the server side determines a target recommended song list based on target information obtained by the display device in the current playback scenario, which can make music recommendations more intelligent and accurate, facilitate the display device side to display the target recommended song list, and thus enhance the user experience.

[0552] Some embodiments of the present application provide a computer-readable non-volatile storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned music recommendation method and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0553] The computer-readable non-volatile storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0554] The present application provides a computer program product, comprising: when the computer program product is run on a computer, the computer is enabled to implement the above-mentioned music recommendation method.

Claims

1. A display device, comprising: a display for displaying images and / or a user interface; Audio player, used to play audio data; Video player, used to play video data and audio data; A user input interface for receiving input signals; a memory configured to store computer instructions and data associated with said display; At least one processor, connected to the display, audio player, user input interface and memory, configured to execute the computer instructions to cause the display device to perform: In response to the received song identification instruction, collecting a first audio segment of the sound data of the current video being played, and sending a song identification request including the first audio segment to a server, so that the server searches for audio data of the sound data based on the first audio segment, wherein the first audio segment represents a segment of the sound data, and the audio data represents audio content similar to or identical to the sound data; If the audio content of the audio data exceeds the audio content range of the sound data, receiving the audio data fed back by the server, controlling the video player to pause playing the current video, recording the first playing progress of the current video when the current video is paused, and controlling the audio player to play the audio data; In response to the received audio stop playback instruction, the audio player is controlled to stop playing the audio data, and the video player is controlled to continue playing the current video based on the first playback progress.

2. The display device according to claim 1 , wherein the processor is further configured to execute computer instructions to cause the display device to perform: If the audio content of the audio data does not exceed the first audio segment of the audio content range of the sound data, the video player is not controlled to pause playing the current video.

3. The display device according to claim 1 , wherein if the audio content of the sound data does not correspond to the end of the audio data, the processor is further configured to execute computer instructions to cause the display device to execute: The audio content similar to or identical to the first audio segment in the audio data is intercepted, and the audio data is played starting from the end point of the intercepted audio content.

4. The display device according to claim 1 , wherein if the audio content of the sound data does not correspond to the end of the audio data, the processor is further configured to execute computer instructions to cause the display device to execute: The audio data is played starting from a starting point of the audio data.

5. The display device according to claim 1 , wherein the processor is further configured to execute computer instructions to cause the display device to perform: Receive the music information of the background music fed back by the server, and control the display of the music information on the display.

6. The display device according to claim 5, wherein the music information includes lyrics of the background music, and the processor is further configured to execute: Controlling the highlighting of lyrics corresponding to the sound data.

7. The display device according to claim 1, wherein the condition that the audio player plays the audio data further comprises that the audio player supports a format of the audio data; If the audio player does not support the format of the audio data, a second prompt message is popped up on the display, wherein the second prompt message is used to prompt the user to install an audio player that supports playing the audio data.

8. The display device according to claim 1 , wherein the processor is further communicatively connected to a microphone, and the processor is further configured to execute computer instructions to cause the display device to: In response to a user request to identify audio, monitoring a playback status of a first application running on the display device; After detecting that the playback state is playing, obtaining first audio data corresponding to the first application within a preset first time range based on the current time, and identifying information of the first audio data by performing a corresponding audio recognition operation on the first audio data; After monitoring that the playback status is not playing, second audio data within a preset second time range is obtained through the microphone, and the information of the second audio data is identified by performing a corresponding audio recognition operation on the second audio data, and the sound source of the second audio data is a sound source outside the display device.

9. The display device according to claim 8, before identifying the information of the first audio data by performing a corresponding audio recognition operation on the first audio data, the processor is further configured to execute computer instructions to cause the display device to perform: monitoring, by the microphone, whether there is audio data emitted by a sound source other than the display device within the preset first time range; After monitoring the audio data emitted by other sound sources other than the display device, the microphone collects third audio data emitted by the other sound sources.

10. The display device according to claim 9, wherein in the step of identifying information of the first audio data by performing a corresponding audio recognition operation on the first audio data, the processor is further configured to execute computer instructions to cause the display device to perform: determining whether the third audio data corresponds to the first audio data; If the third audio data corresponds to the first audio data, performing corresponding audio recognition operations on the first audio data and the third audio data to identify information of the first audio data and the third audio data; If the third audio data does not correspond to the first audio data, information of the first audio data is identified by performing a corresponding audio recognition operation on the first audio data.

11. The display device according to any one of claims 8 to 10, wherein the processor is further configured to execute computer instructions to cause the display device to execute: The audio data to be recognized is delivered to the recognition audio server, wherein the audio data to be recognized includes: first audio data, or second audio data, or first audio data and third audio data; According to the return result of the recognition audio server, the recognition result corresponding to the audio data to be recognized is displayed in the result display area of ​​the display.

12. The display device according to claim 9, wherein the first audio data is data before audio decoding by the display device; The second audio data and the third audio data are data obtained by audio encoding and audio decoding of a sound source outside the display device.

13. The display device according to claim 8, wherein after detecting that the playing state is playing, the processor is further configured to execute computer instructions to cause the display device to execute: If the playback state of the first application is a first playback state, determining to obtain a first recording strategy corresponding to the first playback state based on a preset recording strategy; If the playback state of the first application is the second playback state, based on the preset recording strategy, determine to obtain the second recording strategy corresponding to the first playback state.

14. The display device according to claim 8, wherein after detecting that the playing state is playing, the processor is further configured to execute computer instructions to cause the display device to execute: If there are multiple first applications being played on the display device, a target application is determined from the multiple first applications based on a preset selection strategy, and the first audio data corresponding to the target application is obtained.

15. The display device according to claim 8, wherein in the step of acquiring the second audio data within a preset second time range through the microphone, the processor is configured to execute computer instructions to cause the display device to execute: Acquiring initial audio data outside the display device within a preset second time range through the microphone; The initial audio data is filtered using a preset audio filtering strategy to determine the second audio data.

16. The display device according to claim 1 , wherein the processor is further configured to execute computer instructions to cause the display device to: Display the current user interface; In response to the received song identification instruction, collecting the sound data being played to generate an audio clip and obtaining content data corresponding to the control in the current user interface, wherein the content data represents the media asset corresponding to the control; sending the audio clip and the content data to a server, so that the server identifies song information corresponding to the sound data according to the audio clip and the content data; Receive and display the identified song information fed back by the server.

17. The display device according to claim 16, wherein the current user interface comprises at least two controls; The obtaining of content data corresponding to the control in the current user interface includes: When an active control exists in the current user interface, first content data of the active control is also obtained, wherein the active control is a control having a dynamic item, and the first content data represents media asset information corresponding to the active control, and does not represent media asset information corresponding to inactive controls, wherein the inactive control is a control having no dynamic item; The sending of the audio clip and the content data to a server so that the server identifies song information corresponding to the sound data according to the audio clip and the content data comprises: sending the audio clip and the first content data to a server, so that the server identifies first song information of the sound data according to the audio clip and the first content data; The receiving and displaying the identified song information fed back by the server includes: Receive and display the first song information fed back by the server.

18. The display device according to claim 16, wherein the current user interface comprises at least two controls; The obtaining of content data corresponding to the control in the current user interface includes: When no active control exists in the current user interface, second content data of an inactive control in the current page is also obtained, wherein the inactive control is a control without a dynamic item, and the second content data represents media asset information corresponding to the inactive control; The sending of the audio clip and the content data to a server so that the server identifies song information corresponding to the sound data according to the audio clip and the content data comprises: sending the audio clip and the second content data to a server, so that the server identifies second song information of the sound data according to the audio clip and the second content data; The receiving and displaying the identified song information fed back by the server includes: Receive the second song information fed back by the server.

19. The display device according to claim 17, wherein the active control is a video playback control, a video name is displayed on the video playback control, the first content data is the video name, and the sound data is the sound data being played by the video playback control; The sending of the audio clip and the first content data to a server so that the server identifies the first song information of the sound data according to the audio clip and the first content data comprises: Sending the audio clip and the video name to a server, so that the server searches for a first configuration file corresponding to a first video with the video name, wherein the first configuration file at least includes all audio data configured with the first video and song information corresponding to the audio data; The first song information fed back by the server is received, wherein the first song information is song information of the audio data matching the audio clip found in the first configuration file.

20. The display device according to claim 17, wherein the active control is a video playback control, a first video line is displayed on the video playback control, the first content data is the first video line, and the sound data is the sound data being played by the video playback control; Sending the audio clip and the first content data to a server so that the server identifies the first song information of the sound data according to the audio clip and the first content data includes: The audio clip and the first video lines are sent to the server. If the server can find a second configuration file of the second video according to the first video lines, wherein the second configuration file at least includes all the audio data of the second video configuration and the song information corresponding to the audio data, the server receives the first song information fed back by the server, wherein the first song information is the audio information found in the second configuration file that matches the audio clip. song information of the audio data.

21. The display device according to claim 20, wherein the step of sending the audio clip and the first content data to a server so that the server identifies the first song information of the sound data based on the audio clip and the first content data further comprises: If the server cannot find the second configuration file of the second video according to the first video lines, continue to send the second video lines to the server, wherein the video frame where the second video lines are located is the next video frame where the first video lines are located; If the server can find a second configuration file for the second video based on the lines of the first video and the second video lines, wherein the second configuration file at least includes all audio data configured by the second video and song information corresponding to the audio data, the first song information fed back by the server is received, wherein the first song information is the song information of the audio data matching the audio clip found in the second configuration file.

22. The display device according to claim 17, wherein the active control is a music player control, and the music player control at least plays scrolling lyrics, and the first content data is the scrolling lyrics; Sending the audio clip and the first content data to a server so that the server identifies the first song information of the sound data according to the audio clip and the first content data includes: The audio clip and the first lyrics text of the scrolling lyrics are sent to a server, wherein the audio clip is played while the scrolling lyrics are playing the first lyrics text, so that the server recognizes the first song information of the sound data according to the first lyrics text and the audio clip.

23. The display device according to claim 18, wherein if there is no active control in the current user interface, the condition for obtaining the second content data of the inactive control in the current page further comprises: there is no active control currently running in the foreground; If there is an active control running in the current foreground, after displaying the active control on the user interface, the first content data of the active control is obtained, and the audio clip and the first content data are sent to the server, so that the server identifies the first song information of the sound data based on the audio clip and the first content data, and receives the first song information fed back by the server.

24. The display device of claim 1 , wherein the processor is further configured to execute computer instructions to cause the display device to: Receive input for starting a music recognition function; In response to the operation of starting the music recognition function, target information corresponding to the current playing scene of the display device is obtained, wherein: The current playback scene includes a movie playback scene or a song playback scene. The target information in different current playback scenes is different. The target information is used to represent the current playback scene and the song played in the current playback scene. Sending a recommendation request to a server, wherein the recommendation request includes: the target information, so that the server obtains a target recommended song list based on the target information, and different target information corresponds to different target recommended song lists; Receive a recommendation response sent by the server, wherein the recommendation response includes: the target recommended song list, and control the display to display the target recommended song list.

25. The display device of claim 24, wherein the processor is further configured to execute computer instructions to cause the display device to: When the current playback scene is a film and television playback scene, based on the film and television playback scene, audio segment information and video identification information corresponding to the film and television playback scene are obtained, wherein: The video identification information is identification information corresponding to the target video played in the film and television playback scene; The recommendation request is sent to the server, where the recommendation request includes: the audio segment information and the video identification information, so that the server obtains the target recommended song list according to the audio segment information and the video identification information.

26. The display device according to claim 25, wherein the processor is further configured to execute computer instructions to cause the display device to: Sending an identification request to the server, the identification request including: the audio clip information; receiving a recognition response sent by the server, wherein the recognition response includes: target audio identification information corresponding to the audio segment information; A song recommendation request is sent to the server, wherein the song recommendation request includes: video identification information and the target audio identification information, so that the server obtains the target recommended song list according to the video identification information and the target audio identification information.

27. The display device of claim 26, wherein the processor is further configured to execute computer instructions to cause the display device to: Sending an identification request to the server, the identification request including: the audio clip information; receiving a recognition response sent by the server, wherein the recognition response includes: target audio identification information corresponding to the audio segment information not being recognized; A song recommendation request is sent to the server, wherein the song recommendation request includes video identification information, so that the server obtains the target recommended song list according to the video identification information.

28. The display device of claim 24, wherein the processor is further configured to execute computer instructions to cause the display device to: When the current playback scene is a song playback scene, based on the song playback scene, obtaining audio identification information corresponding to the song playback scene; Sending the recommendation request to the server, wherein the recommendation request includes: The audio identification information is used to enable the server to obtain the target recommended song list based on the audio identification information.

29. The display device of claim 24, wherein the processor is further configured to execute computer instructions to cause the display device to: When the current playback scene is a film and television playback scene, first target audio segment information and first channel identification information corresponding to the film and television playback scene are obtained based on the film and television playback scene, wherein: The first channel identification information is the channel identification information corresponding to the film and television playback scenario; Sending the recommendation request to the server, wherein the recommendation request includes: the first target audio segment information and the first channel identification information, so that the server obtains the target recommended song list according to the first target audio segment information and the first channel identification information; or, When the current playback scene is a song playback scene, based on the song playback scene, obtaining second target audio segment information and second channel identification information corresponding to the song playback scene, wherein the second channel identification information is the channel identification information corresponding to the song playback scene; The recommendation request is sent to the server, where the recommendation request includes: the second target audio segment information and the second channel identification information, so that the server obtains the target recommended song list according to the second target audio segment information and the second channel identification information.

30. The display device according to claim 24, wherein the display comprises a first area and a second area when showing the target recommended song list; The processor is further configured to execute computer instructions to cause the display device to: When a target song corresponding to the target information exists in the target recommended song list, controlling the first area to display information corresponding to the target song, and controlling the second area to display information corresponding to the remaining recommended songs in the target recommended song list except the target song; or, When there is no target song corresponding to the target information in the target recommended song list, the first area is controlled to display prompt information that the target song is not matched, and the second area is controlled to display information corresponding to the recommended songs in the target recommended song list.

31. A method for playing media resources, comprising: In response to the received song recognition instruction, the first audio segment of the sound data of the current video is collected and the audio segment is included Sending a song identification request including the first audio segment to a server, so that the server searches for audio data of the sound data according to the first audio segment, wherein the first audio segment represents a segment of the sound data, and the audio data represents audio content similar to or identical to the sound data; If the audio content of the audio data exceeds the audio content range of the sound data, receiving the audio data fed back by the server, controlling the video player to pause playing the current video, recording the first playing progress of the current video when the current video is paused, and controlling the audio player to play the audio data; In response to the received audio stop playback instruction, the audio player is controlled to stop playing the audio data, and the video player is controlled to continue playing the current video based on the first playback progress.

32. The media asset playing method according to claim 31, further comprising: If the audio content of the audio data does not exceed the audio content range of the sound data, the video player is not controlled to pause playing the current video.

33. The media asset playing method according to claim 31, further comprising: If the audio content of the sound data does not correspond to the end of the audio data, intercepting audio content in the audio data that is similar to or identical to the first audio segment, and starting to play the audio data from the end point of the intercepted audio content as a starting point; If the audio content of the sound data does not correspond to the end of the audio data, the audio data is played from the start point of the audio data.

Images