Display device and audio recognition method

Abstract

A display device and an audio recognition method, the recognition method comprising: in response to an audio recognition request input by a user, establishing communication with a first application program that is playing audio (310), acquiring historical audio data that has been played before a current playing time from a media asset cache corresponding to the first application program (320), a recognition result of the to-be-recognized audio data is determined by performing a recognition operation on the to-be-recognized audio data corresponding to the historical audio data (330), and the recognition result is displayed in an audio recognition display area of the display (340). By performing audio recognition on the historical audio data corresponding to the to-be-recognized audio, the time of waiting for recording by a user is shortened, the audio recognition efficiency is improved, and the user experience is improved.

Description

Display device and audio recognition method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the national patent applications filed with the China Patent Office on December 19, 2022, with application number 202211635428.4, and filed with the China Patent Office on December 19, 2022, with application number 202211634760.9, and filed with the China Patent Office on December 20, 2022, with application number 202211649279.7, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of display technology, and in particular to a display device and an audio recognition method. Background Art

[0004] With the advancement of technology, display devices are becoming increasingly diverse, offering users a wider range of functions. These devices, including smart TVs, smartphones, laser projectors, and other products with display screens, offer users diverse entertainment features such as video, audio, gaming, and karaoke, satisfying their diverse entertainment needs.

[0005] In some display device applications, if a user is interested in the audio played by the display device, they can obtain relevant information about the audio through methods such as audio recognition controls. In the audio recognition method, after receiving a user input request for audio recognition, the display device begins recording audio data and performs audio recognition based on the audio data.

[0006] In audio recognition methods, the longer the recorded audio data, the greater the possibility of its recognition. Therefore, it is often necessary to record audio for a longer time to obtain more accurate recognition results, which requires users to wait for a long time and leads to a poor user experience.

[0007] Summary of the Invention

[0008] In a first aspect, a display device according to an embodiment of the present application includes a display configured to display an image and / or a user interface, a memory configured to store computer instructions and data associated with the display, and at least one processor connected to the display and the memory, configured to execute the computer instructions so that the display device: responds to an audio recognition request input by a user, establishes communication with a first application that is playing audio, obtains historical audio data that has been played before the current playback moment from a media cache corresponding to the first application; determines a recognition result of the audio data to be recognized by performing a recognition operation on audio data to be recognized corresponding to the historical audio data, and displays the recognition result in an audio recognition display area of ​​the display.

[0009] In the second aspect, an audio recognition method according to an embodiment of the present application includes: in response to an audio recognition request input by a user, establishing communication with a first application that is playing audio, and obtaining historical audio data that has been played before the current playback moment from the media resource cache corresponding to the first application; determining the recognition result of the audio data to be recognized by performing a recognition operation on the audio data to be recognized corresponding to the historical audio data, and displaying the recognition result in the audio recognition display area of ​​the display. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG1 illustrates an operation scenario between a display device and a control apparatus according to some embodiments;

[0011] FIG2 is a block diagram of a hardware configuration of a control device 100 according to some embodiments;

[0012] FIG3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments;

[0013] FIG4 is a diagram illustrating software configuration in a display device according to some embodiments;

[0014] FIG5 is a timing diagram of audio recognition in a display device according to some embodiments;

[0015] FIG6 is a schematic diagram illustrating a user's request for a display device to recognize audio according to some embodiments;

[0016] FIG7 a is a schematic diagram of data in a media asset cache according to some embodiments;

[0017] FIG7 b is a schematic diagram of data in another media asset cache according to some embodiments;

[0018] FIG7c is a schematic diagram of data in another media asset cache according to some embodiments;

[0019] FIG7 d is a schematic diagram of data in another media asset cache according to some embodiments;

[0020] FIG7e is a schematic diagram of data in another media asset cache according to some embodiments;

[0021] FIG7f is a schematic diagram of data in yet another media asset cache according to some embodiments;

[0022] FIG8 is a timing diagram of audio recognition in yet another display device according to some embodiments;

[0023] FIG9 is a timing diagram of audio recognition in yet another display device according to some embodiments;

[0024] FIG10 is a timing diagram of a display device performing an audio recognition operation through a recognition audio server according to some embodiments;

[0025] FIG11 is a timing diagram of an audio recognition method in a server according to some embodiments;

[0026] FIG12 is a timing diagram of yet another audio recognition method in a server according to some embodiments;

[0027] FIG13 is a timing diagram of yet another audio recognition method in a server according to some embodiments;

[0028] FIG14 is a timing diagram of an audio recognition method in a display device according to some embodiments;

[0029] FIG15 is a flowchart illustrating initial audio data determination according to some embodiments;

[0030] FIG16 is a flowchart illustrating inter-audio data determination according to some embodiments;

[0031] FIG17 is a timing diagram of an audio recognition method between a display and a server according to some embodiments;

[0032] FIG18 is a schematic diagram of a user interface of a display device according to some embodiments;

[0033] FIG19 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0034] FIG20 is a schematic diagram of a user interface of another display device according to some embodiments;

[0035] FIG21 is a schematic diagram of a media asset playback system architecture according to some embodiments;

[0036] FIG22 is a schematic diagram of a user interface of another display device according to some embodiments;

[0037] FIG23 is a schematic diagram illustrating a principle of the relationship between sound data and audio data according to some embodiments;

[0038] FIG24 is a schematic diagram showing a principle diagram of the relationship between sound data and audio data according to some embodiments;

[0039] FIG25 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0040] FIG26 is a schematic diagram illustrating another principle diagram of the relationship between sound data and audio data according to some embodiments;

[0041] FIG27 is a schematic diagram illustrating another principle of the relationship between sound data and audio data according to some embodiments;

[0042] FIG28 is a signaling diagram of a method for playing media assets according to some embodiments;

[0043] FIG29 is a signaling diagram of another method for playing media assets according to some embodiments;

[0044] FIG30 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0045] FIG31 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0046] FIG32 is a schematic diagram of a user interface of yet another display device according to some embodiments;

[0047] FIG33 is a flow chart of a method for playing media assets according to some embodiments. DETAILED DESCRIPTION

[0048] In order to make the purpose, implementation mode and advantages of the exemplary embodiments of the present application clearer, the exemplary embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0049] Based on the exemplary embodiments shown in this application, all other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of this application. In addition, although the disclosure in this application is described based on one or more exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete embodiment on its own.

[0050] The display device provided in the embodiments of the present application can have various implementation forms, such as a television, smart TV, mobile terminal, tablet computer, computer, laptop computer, laser projection device, monitor, electronic bulletin board, electronic table, etc. Figures 1 and 2 illustrate a specific embodiment of the display device of the present application.

[0051] FIG1 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. As shown in FIG1 , a user can operate a display device 200 through a smart device 300 or a control device 100 .

[0052] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0053] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 can be controlled using an application running on the smart device.

[0054] In some embodiments, the display device may not use the aforementioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0055] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the display device 200 can directly receive the user's voice command control through a module for obtaining voice commands configured inside the display device 200, or it can receive the user's voice command control through a voice control device set outside the display device 200.

[0056] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0057] Figure 2 illustrates a block diagram of the configuration of a control device 100 according to an exemplary embodiment. As shown in Figure 2 , the control device 100 includes a processor 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 receives user input commands and converts them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.

[0058] 3 , the display apparatus 200 includes at least one of a tuner and demodulator 210 , a communicator 220 , a detector 230 , an external device interface 240 , at least one processor 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and a user interface.

[0059] In some embodiments, the at least one processor includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0060] The display 260 includes a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0061] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0062] Communicator 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. Display device 200 can use communicator 220 to send and receive control signals and data signals with control device 100 or server 400.

[0063] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.).

[0064] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0065] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0066] The tuner-demodulator 210 receives broadcast television signals via a wired or wireless reception method, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0067] In some embodiments, the processor 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.

[0068] Processor 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. Processor 250 controls the overall operation of display device 200. For example, in response to receiving a user command to select a UI object to be displayed on display 260, processor 250 may perform operations related to the object selected by the user command.

[0069] In some embodiments, at least one processor includes a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0070] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.

[0071] As shown in Figure 4, in some embodiments, the system is divided into four layers, from top to bottom: the application layer (referred to as the "application layer"), the application framework layer (referred to as the "framework layer"), the Android runtime and system library layer (referred to as the "system runtime library layer"), and the kernel layer.

[0072] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0073] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0074] As shown in Figure 4, the application framework layer in the embodiment of the present application includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (Activity Manager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application package currently installed on the device; a notification manager (Notification Manager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers and desktop components on the user interface.

[0075] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0076] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0077] In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG4 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power driver.

[0078] During the use of the display device, if the user is interested in the audio played by the display device, the user can send a recognition request through the audio recognition control, etc. After receiving the request to recognize the audio input by the user, the display device starts recording audio data and performs audio recognition through the audio data. In order to ensure the probability of recognition, audio data of a certain length needs to be recorded, and the longer the recorded audio data, the greater the possibility of it being recognized. Therefore, the user needs to wait for the audio of the corresponding length to be recorded before a more accurate recognition result can be obtained, which causes the user to wait for a long time.

[0079] In order to reduce the time users wait for recording during the identification process, a display device and audio recognition method according to an embodiment of the present application can establish communication with a first application that is playing audio in response to an audio recognition request input by the user, obtain historical audio data that has been played before the current playback moment from the media resource cache corresponding to the first application, and then determine the information of the audio data to be identified by performing an identification operation on the audio data to be identified corresponding to the historical audio data. The audio to be identified is identified through the historical audio data corresponding to the audio, thereby reducing the time users wait for recording, improving the efficiency of identifying audio, and improving user experience.

[0080] FIG5 is a timing diagram of audio recognition in a display device according to some embodiments. As shown in FIG5 , the display device includes a display and a processor, the processor being communicatively connected to the display, wherein the processor is configured to perform the following steps:

[0081] Play media assets including audio to the server based on the user's playback request;

[0082] Download and cache the audio and video data corresponding to the wall-mounted parsed media assets for media playback.

[0083] S310: In response to an audio recognition request input by a user, establish communication with a first application that is playing audio.

[0084] The user's request for audio recognition may be issued by the user by clicking on the recognition control on the display device 200 , or by inputting by the user through buttons on the control apparatus 100 , voice input, and the like.

[0085] Figure 6 is a schematic diagram of a user's request for a display device to recognize audio according to some embodiments. As shown in Figure 6, the display device has a recognition control 261, and the user operates the recognition control 261 through a control device, etc., triggering a request to recognize audio.

[0086] In response to a user's request to identify audio, first, communication is established with a first application that is playing the audio, so as to facilitate establishing communication with a media asset cache corresponding to the first application.

[0087] As shown in FIG5 , the processor is configured to execute the following steps: S320 , obtaining historical audio data played before the current playing moment from the media resource cache corresponding to the first application.

[0088] It should be understood that the media asset buffer may include audio data and video data.

[0089] Among them, historical audio data is audio data that has been played by the first application before the current playback time. When the user is using the display device and is interested in the audio played by it, the audio is in a playing state. When the user realizes that he wants to know the relevant information of the audio through audio recognition controls, etc., the audio has been played for a period of time. Therefore, when the user issues an audio recognition request through audio recognition controls, etc., the historical audio data corresponding to the audio is already available in the media resource cache corresponding to the first application.

[0090] FIG7 a is a schematic diagram of data in a media asset cache according to some embodiments. As shown in FIG7 a , the data before the current playing moment in the media asset cache is historical audio data that has been played.

[0091] The length of the historical audio data obtained from the media asset cache each time may be different audio data.

[0092] For example, if the first application stores 8 seconds of played audio data in its media cache at the current playback moment, that is, after the user just clicks the audio recognition control, 8 seconds of historical audio data can be obtained from the media cache; if the first application stores 5 seconds of played audio data in the media cache, that is, after the user just clicks the audio recognition control, 5 seconds of historical audio data can be obtained from the media cache; if the first application stores 2 seconds of played audio data in the media cache, that is, after the user just clicks the audio recognition control, 2 seconds of historical audio data can be obtained from the media cache.

[0093] It should be understood that audio recognition can be performed on the audio data to be recognized including the historical audio data obtained in step 320. As shown in FIG5 , the processor is configured to perform the following steps:

[0094] S330 : Determine a recognition result of the audio data to be recognized by performing a recognition operation on the audio data to be recognized corresponding to the historical audio data.

[0095] Perform a recognition operation on the audio data to be recognized corresponding to the historical audio data, wherein the recognition operation can be implemented on the display device side, or the audio data to be recognized can be sent to the server side and the return result of the audio server is recognized.

[0096] It should be understood that there are many types of audio data to be identified. Generally speaking, the types of identification implemented on the server side are more comprehensive and the recognition efficiency is faster. In some embodiments, the identification of audio data to be identified can also be achieved through local data of the display device.

[0097] The capacity of the media cache for storing historical audio data is limited, and the length of the historical audio data may be long or short. Therefore, the length of the audio data to be recognized corresponding to the historical audio data is also limited. When performing audio recognition through the audio data to be recognized, the recognition result of the audio data to be recognized may be recognized information or unrecognized. For the case of unrecognized information, the length of the audio data can be increased before further recognition.

[0098] In some embodiments, since the duration of the historical audio data obtained from the media asset cache is uncertain, before performing the recognition operation, the historical audio data can be further judged to determine whether information such as the duration of the historical audio data meets the preset recognition conditions.

[0099] For example, by identifying statistics of audio data, etc., it is determined that the minimum duration of the audio data to be identified in audio recognition is 5 seconds. At this time, if the historical audio data obtained is greater than or equal to 5 seconds, the audio data to be identified corresponding to the historical audio data can perform the recognition operation; if the historical audio data obtained is less than 5 seconds, the audio data to be identified corresponding to the historical audio data cannot obtain the corresponding recognition result. At this time, the duration of the audio data to be identified can be increased by recording the audio data of the display device after the current playback time, or obtaining the end audio data downloaded for playback after the current playback time from the media resource cache, etc., so as to further realize the recognition of historical audio data less than 5 seconds.

[0100] By increasing the determination of whether historical audio data meets the preset recognition conditions, the efficiency of audio recognition can be improved, and the probability of audio data being recognized can also be increased.

[0101] S340: Display the recognition result of the audio data to be recognized in the audio recognition display area of ​​the display.

[0102] Regarding the display of the recognition results of the audio data to be recognized, if the recognition result is the information of the audio data to be recognized, it can be directly displayed in the result display area, or it can be indirectly displayed in the result display area through a QR code or the like; in some embodiments, it can also be displayed through voice information or the like.

[0103] If the recognition result is unrecognized, information such as recognition result failure may be displayed in the audio recognition display area, or the user may be prompted to perform further recognition.

[0104] The embodiments of the present application reduce the time users spend waiting for recording, improve the efficiency of audio recognition, and enhance user experience by identifying historical audio data corresponding to the audio data to be recognized.

[0105] It should be understood that the duration of the historical audio data is uncertain. After obtaining the historical audio data played before the current playback time from the media asset cache corresponding to the first application in step 320, the process may also include recording the audio data currently being played by the first application. FIG8 is a timing diagram of audio recognition in another display device according to some embodiments. As shown in FIG8, the processor in the display device is further configured to perform the following steps:

[0106] S410: In response to an audio recognition request input by a user, establish communication with a first application that is playing audio.

[0107] S420: Obtain historical audio data played before the current playback moment from the media resource cache corresponding to the first application.

[0108] Among them, historical audio data is audio data that has been played by the first application before the current playback moment. When the user is using the display device and is interested in the audio played by it, the audio is in a playing state. When the user realizes that he wants to know the relevant information of the audio through audio recognition controls, etc., the audio has been played for a period of time. Therefore, when the user issues an audio recognition request through audio recognition controls, etc., the historical audio data corresponding to the audio is already in the media resource cache corresponding to the first application.

[0109] In some embodiments, the audio data in the media cache is stored in segments and is continuously updated in segments. The media cache may contain the first segment data that has been played and the second segment data that is currently being played, or may contain only the second segment data that is currently being played.

[0110] The length of the first segment data and the length of the second segment data may be the same or different.

[0111] For example, the data of the media assets in the display device is stored in the media asset buffer via TS (Transport Stream, an audio and video encapsulation format) segments, and is stored in the form of an audio stream.

[0112] Figure 7b is a schematic diagram of data in another media cache according to some embodiments. As shown in Figure 7b, when there are first segment data that have been played and second segment data that are currently being played in the media cache, there is historical audio data that has been played before the current playback moment in the media cache, and the historical audio data includes the first sub-segment data that has been played before the current playback moment in the second segment data that is being played, and the first segment data that has been played before the second segment data.

[0113] The second segment data may include the first sub-segment data that has been played before the current playback moment and the second sub-segment data that has not been played after the current playback moment.

[0114] In some embodiments, the second segment data may also include the first sub-segment data that has been played before the current playback moment, or the second segment data may include the second sub-segment data that has not been played after the current playback moment.

[0115] Figure 7c is a schematic diagram of data in another media cache according to some embodiments. As shown in Figure 7c, when only the second segment data currently being played exists in the media cache, and the second segment data includes the first sub-segment data that was played before the current playback moment, historical audio data that was played before the current playback moment is in the media cache, and the historical audio data includes the first sub-segment data that was played before the current playback moment in the second segment data being played.

[0116] When the data in the media asset cache is as shown in FIG7a , step 430 may be executed simultaneously with or after step 420 :

[0117] As shown in FIG8 , the processor is further configured to execute the following steps: S430 , within a first preset time period after receiving the audio recognition request, record the audio data played by the first application.

[0118] It should be understood that during the audio recognition process, historical audio data can be obtained through the media cache. At this time, the recorded data played by the first application is recorded. The recorded data is the data played by the first application within the first preset time end after the audio recognition request. The recorded recording data is combined with the historical audio data to increase the length of the audio data to be recognized and improve the recognition efficiency.

[0119] In existing audio recognition, the display device will only start recording audio data after receiving a request for audio recognition input by the user. The embodiment of the present application records audio data on the basis of historical audio data (that is, a part of the audio data is already available), thereby increasing the length of the audio data to be recognized, and controlling the length of the recorded data to be shorter than the length of the audio data recorded in the existing audio recognition through the length of the historical audio data, thereby reducing the length of time the user needs to wait when recording audio data during the audio recognition process.

[0120] For example, in the existing audio recognition, the display device will start recording audio data only after receiving the request for audio recognition input by the user. If the recording time of the audio data takes 15 seconds, the user needs to wait for 15 seconds before the audio recognition function can be performed; while for the embodiment of the present application, after receiving the request for audio recognition input by the user, the display device obtains historical audio data from the media cache. If the duration of the historical audio data is 10 seconds, the corresponding first preset time may take 5 seconds. That is to say, the user only needs to wait for 5 seconds of recording to perform the audio recognition function, thereby reducing the length of time the user needs to wait when recording audio data during the audio recognition process.

[0121] The first preset time is a minimum time that is less than the time required for audio recognition to record audio.

[0122] In some embodiments, a determination of historical audio data may be added before step 430. Since the duration of the historical audio data obtained from the media cache is uncertain, it is possible to further determine whether information such as the duration of the historical audio data meets preset recognition conditions.

[0123] For example, by identifying statistics of audio data, etc., it is determined that the minimum duration of the audio data to be identified in audio recognition is 10 seconds. At this time, if the historical audio data obtained is greater than or equal to 10 seconds, the audio data to be identified corresponding to the historical audio data can be identified through step 330. The recognition operation can be performed; if the historical audio data obtained is less than 10 seconds, the audio data to be identified corresponding to the historical audio data cannot obtain the corresponding recognition result. At this time, the duration of the audio data to be identified can be increased by recording the audio data of the display device after the current playback time through step 430, thereby further realizing the recognition of the corresponding audio data (i.e., the recorded data and the historical audio data).

[0124] By increasing the determination of whether historical audio data meets the preset recognition conditions, the efficiency of audio recognition can be improved, and the probability of audio data being recognized can also be increased.

[0125] It should be understood that when the duration of the audio data to be identified is not sufficient for audio recognition, the audio data to be identified can be further expanded by adding step 430, that is, step 430 records the recording data played by the first application within the first preset time period after receiving the audio recognition request; when the duration of the audio data to be identified meets the requirements of audio recognition or meets the preset recognition conditions, step 430 does not need to be executed.

[0126] As shown in Figure 7b, if the historical audio data includes the first sub-segment data of the second segment data being played before the current playback moment, step 430 can be configured to obtain the first alternative segment data played by the first application within the second preset time after the current playback moment of the historical audio data, and the recording data includes the first alternative segment data.

[0127] In step 430, the first preset time period may be a preset fixed time length or a dynamically adjusted time length.

[0128] In some embodiments, a preset recognition duration can be set, which is the minimum time that audio data can be recognized, and the duration of the audio data recorded for the first application can be determined by comparing the duration of the first sub-segment data with the preset recognition duration.

[0129] If the duration of the first sub-segment data is less than the preset identification duration, a first playback duration (i.e., the first preset time period in step 430) is determined, where the first playback duration is the difference between the preset identification duration and the duration of the first sub-segment data;

[0130] First candidate segment data played by a first application within a first playback duration after a current playback moment of the historical audio data is obtained, where the recording data includes the first candidate segment data.

[0131] If the duration of the first sub-segment data is not less than the preset recognition duration, recognition can be performed using the audio data to be recognized corresponding to the first sub-segment data.

[0132] As shown in Figure 7c, if the historical audio data includes the first sub-segment data that has been played before the current playback moment in the second segment data being played, and the first segment data that has been played before the second segment data, step 430 can be configured to obtain the second alternative segment data played by the first application within a third preset time after the current playback moment of the historical audio data, and the recorded data includes the second alternative segment data.

[0133] In some embodiments, a preset recognition time period may be set, which is the minimum time during which audio data can be recognized, and the duration of the played data may be compared with the preset recognition time period, wherein the duration of the played data is the sum of the duration of the first sub-segment data and the duration of the first segment data;

[0134] If the duration of the played data is less than the preset identification duration, a second play duration (i.e., the first preset time period in step 430) is determined, where the second play duration is the difference between the preset identification duration and the duration of the played data;

[0135] Second candidate segment data played by the first application within a second playback duration after the current playback moment of the historical audio data is obtained, and the recording data includes the second candidate segment data.

[0136] If the duration of the played data is not less than the preset recognition duration, recognition can be performed using the first sub-segment data and the audio data to be recognized corresponding to the first segment data.

[0137] S440: Determine a recognition result of the audio data to be recognized by performing a recognition operation on the audio data to be recognized, wherein the audio data to be recognized includes historical audio data and recorded data.

[0138] Among them, if the historical audio data is as shown in Figure 7b, the audio data to be identified includes the first segment data, the first sub-segment data and the recording data; if the historical audio data is as shown in Figure 7c, the audio data to be identified includes the first sub-segment data and the recording data.

[0139] Regardless of whether the audio data to be identified includes historical audio data, or the audio data to be identified includes historical audio data and recorded data; the recognition operation can be implemented on the display device side, or the audio data to be identified can be sent to the server side and the audio server returns the result based on the identification.

[0140] It should be understood that there are many types of audio data to be identified. Generally, identification is implemented on the server side. In some embodiments, identification of the audio data to be identified can also be implemented through local data of the display device.

[0141] S450: Display the recognition result of the audio data to be recognized in the audio recognition display area of ​​the display.

[0142] Regarding the display of the recognition results of the audio data to be recognized, if the recognition result is the information of the audio data to be recognized, it can be directly displayed in the result display area, or it can be indirectly displayed in the result display area through a QR code or the like; in some embodiments, it can also be displayed through voice information or the like.

[0143] The embodiments of the present application reduce the time users spend waiting for recording, improve the efficiency of audio recognition, and enhance user experience by identifying historical audio data or historical audio data and recording data corresponding to the audio data to be recognized.

[0144] It should be understood that the media resource cache may include not only historical audio data that has been played, but also the last audio data that has been downloaded but not played. The historical audio data that has been played before the current playback time and the last audio data that has been downloaded for playback after the current playback time can be obtained from the media resource cache corresponding to the first application. FIG9 is a timing diagram of audio recognition in another display device according to some embodiments. As shown in FIG9, the processor in the display device is further configured to perform the following steps:

[0145] S510: In response to an audio recognition request input by a user, establish communication with a first application that is playing audio.

[0146] S520: Determine whether there is tail audio data in the media resource cache corresponding to the first application.

[0147] It should be understood that, during the use of the first application, its media resource cache may include historical audio data that has been played, and may also include historical audio data and downloaded but unplayed last audio data.

[0148] In some embodiments, the end audio data may be determined by parsing unplayed data in the media asset cache, or may be determined by parsed unplayed data in the media asset cache.

[0149] In some embodiments, the end audio data may be determined by converting the parsed digital audio data.

[0150] If there is historical audio data and last audio data in the media asset cache, the historical audio data and last audio data can be obtained, the audio data to be identified corresponding to the historical audio data and last audio data can be determined, the length of the audio data to be identified can be increased, and the probability of the audio data to be identified being identified can be improved.

[0151] S530: Obtain historical audio data played before the current playback time and last audio data after the current playback time from the media resource cache.

[0152] The last audio data refers to the data that has been downloaded and not played after the current playing time.

[0153] Figure 7d is a schematic diagram of data in another media cache according to some embodiments. As shown in Figure 7d, when there is historical audio data that has been played and the last audio data that has been downloaded but not played in the media cache, the last audio data is the audio data that has been downloaded but not played after the current playback moment.

[0154] Figure 7e is a schematic diagram of data in another media cache according to some embodiments. As shown in Figure 7e, when there are historical audio data that have been played and downloaded but not played last audio data in the media cache, the last audio data is the audio data that has been downloaded but not played after the current playback moment, and the last audio data includes the second sub-segment data of the second segment data being played, which has been downloaded but not played after the current playback moment.

[0155] Figure 7f is a schematic diagram of data in another media cache according to some embodiments. As shown in Figure 7f, when there are historical audio data that have been played and downloaded but not played last audio data in the media cache, the last audio data is the audio data that has been downloaded but not played after the current playback moment, and the last audio data includes the second segment data being played, the second sub-segment data that has been downloaded but not played after the current playback moment, and the third segment data that has been downloaded but not played after the second segment data.

[0156] It should be understood that for the historical audio data in Figures 7d, 7e and 7f, the historical audio data may include the first sub-segment data of the second segment data being played before the current playback moment, or the historical audio data may also include the first segment data and the first sub-segment data played before the second segment data.

[0157] That is to say, the historical audio data and the last audio data in the media asset cache can be any combination of the historical audio data in FIG. 7 b and FIG. 7 c and the last audio data in FIG. 7 e and FIG. 7 f .

[0158] S540: Determine a recognition result of the audio data to be recognized by performing a recognition operation on the audio data to be recognized, wherein the audio data to be recognized includes historical audio data and end audio data.

[0159] Among them, if the end audio data is as shown in Figure 7e, the audio data to be identified includes historical audio data and the second sub-segment data; if the historical audio data is as shown in Figure 7c, the audio data to be identified includes historical audio data, the second sub-segment data and the third segment data.

[0160] For example, in the media cache, the historical audio data is 8 seconds and the last audio data is 8 seconds. The corresponding audio data to be identified is 16 seconds. At this time, after responding to the audio recognition request input by the user, the historical audio data and the last audio data in the media cache are obtained for a total of 16 seconds, and audio recognition can be performed through the historical audio data and the audio data to be identified corresponding to the last audio data, so that the user does not need to wait for the audio recording time, which improves the speed of audio recognition and improves the user experience.

[0161] Regardless of whether the audio data to be identified includes historical audio data, or the audio data to be identified includes historical audio data and the end audio data; the recognition operation can be implemented on the display device side, or the audio data to be identified can be sent to the server side and the return result of the audio server is recognized.

[0162] It should be understood that there are many types of audio data to be identified. Generally, identification is implemented on the server side. In some embodiments, identification of the audio data to be identified can also be implemented through local data of the display device.

[0163] S550: Display the recognition result of the audio data to be recognized in the audio recognition display area of ​​the display.

[0164] Regarding the display of the recognition results of the audio data to be recognized, if the recognition result is the information of the audio data to be recognized, it can be directly displayed in the result display area, or it can be indirectly displayed in the result display area through a QR code or the like; in some embodiments, it can also be displayed through voice information or the like.

[0165] FIG10 is a timing diagram of an audio recognition operation performed by a display device through an audio recognition server according to some embodiments. As shown in FIG10 , the audio data to be recognized may be determined based on different audio data, specifically including the following steps:

[0166] S601: Determine audio data to be recognized.

[0167] Based on the timing diagrams of Figures 5-6, 7a-7f, and 8, it should be understood that the audio data to be recognized includes: first audio data, or second audio data, or third audio data, or first audio data and third audio data.

[0168] S602: The audio data to be recognized is transmitted to the recognition audio server.

[0169] On the audio recognition server side, recognition is performed on the audio data to be recognized.

[0170] In some embodiments, the type of audio data to be identified can be reduced to a preset data type on the identification audio server. After the type is reduced, the amount of audio data to be identified is smaller, the speed of transmission to the identification audio server is faster, and the efficiency of identification by the identification audio server is also improved. Therefore, before step 520, the type of audio data to be identified can be reduced to a preset data type on the identification audio server.

[0171] S603: Receive the recognition result sent by the audio recognition server, and display the recognition result on a display device.

[0172] The information of the recognized audio data to be recognized (recognition results) can be displayed directly in the result display area, or indirectly in the result display area through a QR code or the like; in some embodiments, it can also be displayed through voice information or the like.

[0173] It should be understood that there are many types of audio data to be identified. Generally, the identification is achieved on the server side as shown in Figure 10. In some embodiments, the identification of the audio data to be identified can also be achieved through local data of the display device.

[0174] The embodiments of the present application reduce the time users spend waiting for recording, improve the efficiency of audio recognition, and enhance user experience by identifying historical audio data or historical audio data and end audio data corresponding to the audio data to be recognized.

[0175] In addition to the above-mentioned embodiments of audio data recognition, this application also provides some other embodiments of audio data recognition, which are described in detail as follows:

[0176] During use of the display device, the user may be interested in the audio played by the display device and may learn relevant information about the audio by pressing an audio recognition button or the like.

[0177] For example, if a user is interested in the music played by the media, he or she may input a request to identify the audio through the audio recognition button. After receiving the request, the display device records the audio data corresponding to the music according to the fixed recording time corresponding to the user's operation of the audio recognition button, and performs audio recognition on the audio data. The result of the audio recognition can only be obtained after the user waits for the audio to be recorded for a fixed recording time.

[0178] For different audio data, the length of audio data required to obtain the recognition result is different. If each audio to be recognized is recorded with a fixed recording time, the user will need to wait for a time greater than or equal to the fixed recording time before obtaining the recognition result when using this function.

[0179] To improve the speed of audio recognition, an embodiment of the present application provides a server, a display device, and an audio recognition method, wherein the server receives initial audio data and data information of the initial audio data from a display device, the data information including an audio coding identifier and a first timing identifier, and the duration of the initial audio data is a preset first duration; monitors whether intermediate audio data from the display device is received, the data information of the intermediate audio data including the same audio coding identifier and second timing identifier as the initial audio data, the duration of the intermediate audio data is a preset second duration, the preset second duration is less than or equal to the preset first duration, and the sum of the preset first duration and the preset second duration is less than the recording start and end duration corresponding to the display device's response to a user's request to recognize audio; if the intermediate audio data is monitored within the preset monitoring duration, the initial audio data and the intermediate audio data are spliced ​​based on the order of the first timing identifier and the second timing identifier to determine the target audio data; wherein the preset monitoring duration is greater than the preset second duration; by performing a corresponding audio recognition operation on the target audio data, information of the target audio data is identified, and by recording the audio to be recognized in segments, the server can perform audio recognition after obtaining partial audio data, thereby improving the speed of audio recognition and enhancing the user experience.

[0180] FIG11 is a timing diagram of an audio recognition method in a server according to some embodiments. As shown in FIG11 , the server includes a processor configured to perform the following steps:

[0181] S301: Receive initial audio data and data information of the initial audio data sent by a display device, where the data information includes an audio coding identifier and a first timing identifier, and the duration of the initial audio data is a preset first duration.

[0182] The initial audio data is audio data used for recognition, and the duration of the initial audio data is a preset first duration.

[0183] In some embodiments, the preset first duration corresponding to the initial audio data can be set to be less than or equal to the minimum duration required to identify the audio data.

[0184] The audio coding identifier in the data information is the unique identifier of the initial audio data. The audio coding identifier is used to represent this audio recognition and exists in the entire process of this audio recognition; the first timing identifier is used to represent the order of the initial audio data in all the audio data that need to be recognized in this audio recognition. It should be understood that the initial audio data is the first segmented data in the audio data of this audio recognition, so the order of its first timing identifier is located first in the audio data of this time.

[0185] In some embodiments, the first timing identifier may be a timestamp or a serial number corresponding to time, that is, it may be a digital serial number used to represent the timing of the initial audio data, or it may be information such as a timestamp used to identify the timing of the initial audio data.

[0186] S302: Monitor whether intermediate audio data is received, where the data information of the intermediate audio data includes an audio coding identifier and a second timing identifier, and the duration of the intermediate audio data is a preset second duration.

[0187] The intermediate audio data is data having the same audio coding identifier as the initial audio data. It should be understood that the intermediate audio data and the initial audio data are data determined in the same audio recognition, and the intermediate audio data and the initial audio data are sent to the server by the same display device.

[0188] It should be understood that the initial audio data and its data information, the intermediate audio data and its data information are determined in one audio recognition of a certain display device.

[0189] If the server receives other audio data (not intermediate audio data), it can be understood that the audio coding identifier of the other audio data is different from the audio coding identifier of the initial audio data. Then the data representing the other audio data and the initial audio data may be data determined by different audio recognition times in the same display device, or may be data determined by different audio recognition times in different display devices.

[0190] By monitoring whether the intermediate audio data is received at the server, it is determined that the audio data to be recognized includes the initial audio data, or includes the initial audio data and the intermediate audio data.

[0191] It should be understood that the number of intermediate audio data may be one or more, and the preset second durations corresponding to the intermediate audio data are the same duration.

[0192] The preset second duration is less than or equal to the preset first duration, and the sum of the preset first duration and the preset second duration is less than the recording start and end durations corresponding to the display device's response to the user's request to recognize audio.

[0193] That is, the duration of the intermediate audio data is less than or equal to the duration of the initial audio data. The start and end duration of the recording corresponding to the user's request for audio recognition in the display device is the maximum time the user needs to wait for the audio recognition.

[0194] In some embodiments, if the duration of the initial audio data satisfies the larger duration that is less than or equal to the minimum duration required to identify the audio data, the duration of the intermediate audio data can be set to be less than the duration of the initial audio data, and the preset second duration corresponding to the intermediate audio data can be relatively small, that is, the intermediate audio data is data determined by a small period. For example, the minimum duration required to identify the audio data is 15 milliseconds, the preset first duration corresponding to the initial audio data is 10 milliseconds, and the preset second duration corresponding to each intermediate audio data is 1 millisecond; it can also be that the preset first duration corresponding to the initial audio data is 15 milliseconds, and the preset second duration corresponding to each intermediate audio data is 1 millisecond.

[0195] In some embodiments, if the duration of the initial audio data satisfies the smaller duration that is less than or equal to the minimum duration required to identify the audio data, the duration of the intermediate audio data can be set equal to the duration of the initial audio data. For example, the minimum duration required to identify the audio data is 15 milliseconds, the preset first duration corresponding to the initial audio data is 3 milliseconds, and the preset second duration corresponding to each intermediate audio data is 3 milliseconds.

[0196] FIG12 is a timing diagram of another audio recognition method in a server according to some embodiments. As shown in FIG12 , after step 301, the following steps may be further performed:

[0197] S410: Identify information of the target audio data by performing a corresponding audio recognition operation on the target audio data corresponding to the initial audio data.

[0198] It should be understood that step 410 may be performed before, simultaneously with, or after step 302 .

[0199] The preset first duration corresponding to the initial audio data may be less than or equal to the minimum duration required to identify the audio data. That is, after the server receives the initial audio data, its corresponding target audio data may be able to identify the result of audio recognition. Therefore, at this time, the corresponding audio recognition operation can be performed through step 410 to identify the information of the target audio data corresponding to the initial audio data, thereby improving the speed of audio recognition and enhancing the user experience.

[0200] As shown in Figure 11, the processor is also configured to perform the following steps: S303. If intermediate audio data is monitored within the preset monitoring time, the initial audio data and the intermediate audio data are spliced ​​based on the order of the first timing identifier and the second timing identifier to determine the target audio data.

[0201] Among them, the preset monitoring time is greater than the preset second time, that is, if the server receives the intermediate audio data sent by the display device, the server can monitor at least one intermediate audio data within the preset monitoring time. It should be understood that multiple intermediate audio data can also be monitored.

[0202] In some embodiments, the preset monitoring duration may also be greater than or equal to the recording start and end durations, that is, within a corresponding recording start and end period, all possible intermediate audio data corresponding to this audio recognition may be monitored.

[0203] The intermediate audio data is the data other than the first segment in the audio data of this audio recognition, that is, the data from the second segment to the Nth segment can all be the intermediate audio data.

[0204] The second timing identifier represents the sequence of the intermediate audio data when it was recorded, so the second timing identifier corresponding to each intermediate audio data is different. The second timing identifier can be a timestamp or a serial number corresponding to time, that is, it can be a digital serial number used to represent the timing of the initial audio data, or it can be information such as a timestamp used to identify the timing of the initial audio data. The second timing identifier and the first timing identifier use the same timing recording standard. For example, the second timing identifier of intermediate audio data A is 2, the second timing identifier of intermediate audio data B is 3, and the second timing identifier of intermediate audio data C is 4; for another example, the second timing identifier of intermediate audio data X is 0.010 seconds, the second timing identifier of intermediate audio data Y is 0.011 seconds, and the second timing identifier of intermediate audio data Z is 0.012 seconds.

[0205] Wherein, based on the order of the first timing identifier and the second timing identifier, the initial audio data and the intermediate audio data are spliced ​​to determine the target audio data.

[0206] Regardless of the timing recording standard adopted, the first timing identifier of the initial audio data is always in the first order. For each intermediate audio data, the initial audio data and the intermediate audio data are spliced ​​according to the order of its corresponding second timing identifier and the sequential relationship between it and the first timing identifier.

[0207] For example, the first timing identifier of the initial audio data is 1, the second timing identifier of the intermediate audio data A received in sequence on the server side is 2, the second timing identifier of the intermediate audio data C is 4, and the second timing identifier of the intermediate audio data B is 3. The corresponding target audio data can be spliced ​​in the order of initial audio data and intermediate audio data A; further spliced ​​in the order of initial audio data, intermediate audio data A, and intermediate audio data B; or spliced ​​in the order of initial audio data, intermediate audio data A, intermediate audio data B, and intermediate audio data C.

[0208] It should be understood that the first timing identifier and the second timing identifier are what the server has when it receives the corresponding audio data (initial audio data, intermediate audio data), not the order in which the audio data arrive at the server. If there are network problems in the data transmission between the display and the server, it may cause confusion in the timing of receiving the intermediate audio data. The first timing identifier and the second timing identifier are used to ensure that the server side splices the initial audio data and the intermediate audio data in chronological order.

[0209] For each monitored intermediate audio data, it is spliced ​​with the previous initial audio data, or the target audio data obtained by splicing the previous initial audio data and the intermediate audio data can be used for audio recognition through the following step 304 until the information of the target audio data is recognized or no intermediate audio data is received.

[0210] As shown in FIG11 , the processor is further configured to perform the following steps: S304 , identifying information of the target audio data by performing a corresponding audio recognition operation on the target audio data.

[0211] The audio recognition operation may be set according to the type of the user's request for audio recognition. When audio recognition is performed on target audio data, the information of the recognized target audio data may be recognized information or unrecognized information.

[0212] By continuously splicing together the received segmented audio data to be recognized (initial audio data and intermediate audio data), the server can realize audio recognition after obtaining partial audio data (initial audio data, or initial audio data and one or more intermediate audio data spliced ​​together), thereby improving the speed of audio recognition and enhancing the user experience.

[0213] FIG13 is a timing diagram of an audio recognition method in a server according to some embodiments. As shown in FIG13 , after step 302 , the following steps may be further performed:

[0214] S305: If no intermediate audio data is detected within the preset monitoring time, information of the target audio data is identified by performing a corresponding audio recognition operation on the target audio data corresponding to the initial audio data.

[0215] The preset first duration corresponding to the initial audio data can be less than or equal to the minimum duration required to identify the audio data. That is to say, after the server receives the initial audio data and has not received the intermediate audio data, its corresponding target audio data may be able to recognize the result of audio recognition. By performing the corresponding audio recognition operation on the target audio data corresponding to the initial audio data, the information of the target audio data can be identified, the speed of audio recognition can be improved, and the user experience can be enhanced.

[0216] FIG14 is a timing diagram of an audio recognition method in a display device according to some embodiments. As shown in FIG14 , the display device includes a display and a processor, the display and the processor are communicatively connected, and the processor is configured to perform the following steps:

[0217] S501: In response to a user's request for audio recognition, obtain initial audio data of a preset first duration and data information corresponding to the initial audio data, where the data information includes an audio coding identifier and a first timing identifier.

[0218] The user's request for audio recognition may be issued by the user by clicking on the recognition control on the display device 200 , or by inputting by the user through buttons on the control apparatus 100 , voice input, and the like.

[0219] Figure 9 is a schematic diagram of a user's request for a display device to recognize audio according to some embodiments. As shown in Figure 9, the display device has a recognition control 261, and the user operates the recognition control 261 through a control device, etc., triggering a request to recognize audio.

[0220] It should be understood that the initial audio data is the first audio data recorded for identification in response to the user's request to identify the audio, and its preset first duration can be set to be less than or equal to the minimum duration required to identify the audio data.

[0221] For example, the minimum duration required for identifying audio data is 15 milliseconds, the preset first duration corresponding to the initial audio data may be 10 milliseconds, and the preset first duration corresponding to the initial audio data may also be 3 milliseconds.

[0222] The data information corresponding to the initial audio data includes an audio coding identifier and a first timing identifier. The audio coding identifier is a unique identifier of the initial audio data. The audio coding identifier is used to represent this audio recognition and exists in the entire process of this audio recognition; the first timing identifier is used to represent the order of the initial audio data in all the audio data that need to be recognized in this audio recognition. It should be understood that the initial audio data is the first segmented data in the audio data of this audio recognition, so the order of its first timing identifier is located first in the audio data of this time.

[0223] In some embodiments, the initial audio data can be re-encoded and decoded. The audio format of the recorded initial audio data that needs to be recognized is generally higher than the audio format required for server recognition. The initial audio data can be re-encoded and decoded to reduce the audio format of the initial audio data, reduce the data volume of the initial audio data, and increase the rate at which the initial audio data is transmitted to the server.

[0224] FIG15 is a flowchart of determining initial audio data according to some embodiments. As shown in FIG15 , in step 501, in response to a user's request to recognize audio, initial audio data is obtained based on a preset first duration, including:

[0225] S511 : In response to a user's request to identify audio, record first audio data based on a preset first duration.

[0226] S512: Encode the first audio data using a preset encoding and decoding strategy to determine initial audio data.

[0227] The audio format of the first audio data is reduced by a preset coding strategy, the data volume of the first audio data is reduced, and the rate at which the initial audio data is transmitted to the server is increased.

[0228] As shown in FIG14 , the processor is configured to perform the following steps: S502 , sending the initial audio data and data information of the initial audio data to the server.

[0229] S503. Obtain intermediate audio data of a preset second duration, where the intermediate audio data includes an audio coding identifier and a second timing identifier; wherein the preset first duration is greater than or equal to the preset second duration, and the sum of the preset first duration and the preset second duration is less than the recording start and end durations corresponding to the request.

[0230] It should be understood that the intermediate audio data is the audio data recorded continuously after the initial audio data. Both the initial audio data and the intermediate audio data record the audio data that needs to be identified this time. Among them, the intermediate audio data is data that is temporally connected to the initial audio data or the previous intermediate audio data, so the intermediate audio data and the initial audio data have the same audio coding identifier.

[0231] The first timing identifier and the second timing identifier may be digital numbers used to represent the timing of the initial audio data, or may be information such as a timestamp used to identify the timing of the initial audio data.

[0232] In some embodiments, the first timing identifier is determined based on the time when the initial audio data is acquired, and the second timing identifier is determined based on the time when the intermediate audio data is acquired.

[0233] The preset first duration is greater than or equal to the preset second duration, that is, the duration of the intermediate audio data is less than or equal to the duration of the initial audio data.

[0234] The recording start and end times corresponding to the user's request for audio recognition in the display device are the longest time the user needs to wait for the audio recognition; among them, the sum of the preset first time and the preset second time is less than the recording start and end times corresponding to the request.

[0235] In some embodiments, if the duration of the initial audio data satisfies the larger duration that is less than or equal to the minimum duration required to identify the audio data, the duration of the intermediate audio data can be set to be less than the duration of the initial audio data, and the preset second duration corresponding to the intermediate audio data can be relatively small, that is, the intermediate audio data is data determined by a small period. For example, the minimum duration required to identify the audio data is 15 milliseconds, the preset first duration corresponding to the initial audio data is 15 milliseconds, and the preset second duration corresponding to each intermediate audio data is 1 millisecond; it can also be that the preset first duration corresponding to the initial audio data is 15 milliseconds, and the preset second duration corresponding to each intermediate audio data is 1 millisecond.

[0236] In some embodiments, if the duration of the initial audio data satisfies the smaller duration that is less than or equal to the minimum duration required to identify the audio data, the duration of the intermediate audio data can be set equal to the duration of the initial audio data. For example, the minimum duration required to identify the audio data is 15 milliseconds, the preset first duration corresponding to the initial audio data is 2 milliseconds, and the preset second duration corresponding to each intermediate audio data is 2 milliseconds.

[0237] In some embodiments, the intermediate audio data can be re-encoded and decoded. The audio format of the recorded intermediate audio data that needs to be recognized is generally higher than the audio format required for server recognition. By re-encoding and decoding the intermediate audio data, the audio format of the intermediate audio data can be lowered, the data volume of the intermediate audio data can be reduced, and the rate at which the intermediate audio data is transmitted to the server can be increased.

[0238] It should be understood that the intermediate audio data and the initial audio data are re-encoded and decoded in the same manner to ensure that the audio format of the target audio data after the server splices the intermediate audio data and the initial audio data is the same.

[0239] FIG16 is a flowchart of determining intermediate audio data according to some embodiments. As shown in FIG16 , obtaining the intermediate audio data based on the preset second duration in step 503 includes:

[0240] S531: Record second audio data based on a preset second duration.

[0241] S532: Encode the second audio data using a preset encoding and decoding strategy to determine intermediate audio data.

[0242] The audio format of the second audio data is reduced by a preset coding and decoding strategy, the data volume of the second audio data is reduced, and the rate at which the intermediate audio data is transmitted to the server is increased.

[0243] As shown in FIG14 , the processor is configured to execute the following steps: S504 , sending the intermediate audio data to the server.

[0244] S505: Receive information for identifying target audio data from the server, and display the information on a display. The target audio data includes initial audio data, or initial audio data and intermediate audio data.

[0245] Regarding the display of the recognition results of the audio data to be recognized, if the recognition result is the information of the audio data to be recognized, it can be directly displayed in the result display area, or it can be indirectly displayed in the result display area through a QR code or the like; in some embodiments, it can also be displayed through voice information or the like.

[0246] The embodiment of the present application records the audio segments to be recognized and sends them to the server, so that the server can realize audio recognition after obtaining partial audio data (initial audio data, or initial audio data and one or more intermediate audio data spliced ​​together), thereby improving the speed of audio recognition and enhancing the user experience.

[0247] FIG17 is a timing diagram of an audio recognition method between a display and a server according to some embodiments. As shown in FIG17 , the server includes a processor, and the display device also includes a processor. The audio recognition method includes the following steps:

[0248] S610: In response to a user request for audio recognition, obtain initial audio data of a preset first duration and data information of the initial audio data, where the data information of the initial audio data includes an audio coding identifier and a first timing identifier.

[0249] It should be understood that the preset first duration can be set to be less than or equal to the minimum duration required for identifying audio data.

[0250] S620: Transmit the initial audio data and data information of the initial audio data.

[0251] It should be understood that the initial audio data and the data information of the initial audio data are sent from the display device to the server.

[0252] After receiving the initial audio data and the data information of the initial audio data, the server may perform a corresponding audio recognition operation according to the target audio data corresponding to the initial audio data to recognize the information of the target audio data.

[0253] On the display device side, step S630 is executed.

[0254] S630. Obtain intermediate audio data of a preset second duration and data information of the intermediate audio data, the data information of the intermediate audio data including an audio coding identifier and a second timing identifier, wherein the preset second duration is less than or equal to the preset first duration, and the sum of the preset first duration and the preset second duration is less than the recording start and end durations corresponding to the display device's response to the user's request to identify audio.

[0255] The intermediate audio data is data that is temporally connected to the initial audio data or the previous intermediate audio data, and the audio coding identifier corresponding to the intermediate audio data is the same as the audio coding identifier corresponding to the initial audio data.

[0256] S640: Transmit the intermediate audio data and data information of the intermediate audio data.

[0257] It should be understood that the intermediate audio data and the data information of the intermediate audio data are transmitted from the display device to the server.

[0258] On the server side, execute step S650.

[0259] S650: After receiving the initial audio data, identify information of the target audio data by performing a corresponding audio recognition operation on the target audio data corresponding to the initial audio data.

[0260] The timing identifier includes a first timing identifier and a second timing identifier. The timing identifier includes: a timestamp or a sequence number corresponding to time.

[0261] S660: Monitor whether the intermediate audio data and the data information of the intermediate audio data are received.

[0262] S671: If intermediate audio data is monitored within the preset monitoring time, the initial audio data and the intermediate audio data are spliced ​​based on the order of the first timing identifier and the second timing identifier to determine the target audio data.

[0263] The preset monitoring duration is greater than the preset second duration.

[0264] S672: If no intermediate audio data is detected within the preset monitoring time, information of the target audio data is identified by performing a corresponding audio recognition operation on the target audio data corresponding to the initial audio data.

[0265] S680: Identify information of the target audio data by performing a corresponding audio recognition operation on the target audio data.

[0266] S690: Receive information for identifying target audio data from the server, and display the information on a display. The target audio data includes initial audio data, or initial audio data and intermediate audio data.

[0267] The embodiment of the present application records the audio segments to be recognized and sends them to the server, so that the server can realize audio recognition after obtaining part of the audio data, thereby improving the speed of audio recognition and enhancing the user experience.

[0268] The embodiment of the present application records the audio segments to be recognized and sends them to the server, so that the server can realize audio recognition after obtaining partial audio data (initial audio data, or initial audio data and one or more intermediate audio data spliced ​​together), thereby improving the speed of audio recognition and enhancing the user experience.

[0269] In addition to the above embodiments regarding audio data recognition, this application also provides other embodiments regarding media resource playback, which are described in detail as follows:

[0270] In the related art of some embodiments, when a user is watching a video using a video player, the video being played may be configured with background music. For example, in the user interface shown in FIG18 , the emotional TV series XX is being played while the lyrical song YY is playing as background music. If the audio playing in the video is only a part of the background music, the user may be interested in the background music and want to play other parts of the background music. The user can manually pause the video playback (for example, click on the video screen to pause the video playback. In the user interface shown in FIG19 , the background music is also paused when the video is paused).

[0271] After that, the user obtains basic information about the background music (for example, by searching extensively online or asking other netizens in forums), and then searches the music software based on the basic information (in the user interface shown in FIG20 ) to hear the rest of the background music. After listening to the rest of the background music, the user returns to the video playback page and manually clicks the continue button of the video playback software to continue playing the video. This method of playing background music has many steps and is complicated, and the user experience is also poor.

[0272] To address the problems in the related art described above, embodiments of the present application provide a method for media playback. This method can be applied to the system shown in FIG21 . As shown in FIG21 , the system may include a server 400 and a display device 200 used by a user. For example, server 400 may be any data processing server, such as a cloud server or a distributed server.

[0273] In the user interface of the video player shown in FIG22 , in addition to displaying the video playback screen, a song identification button is also provided. When the user clicks the song identification button, the display device generates a song identification instruction in response to the click of the song identification button. In this case, the video player does not pause the currently playing video. After receiving the song identification instruction, the display device 200 begins recording the audio data of the currently playing video.

[0274] In some embodiments, a song identification button may not be set on the playback screen to avoid affecting the user's viewing experience. The user may trigger the display device to generate a song identification instruction by inputting a preset voice command or by using a preset case of a remote control device to enable the display device to generate a song identification instruction.

[0275] In some embodiments, when certain videos (e.g., TV series or movies) are played with background music, the audio of the characters in the video is often played simultaneously with the background music. To improve the accuracy of song identification, the audio data of the characters speaking can be removed from the recorded audio. The specific implementation of removing the audio data of the characters speaking is not described in detail in this application.

[0276] In some embodiments, while extracting a first audio segment of background music from the audio data of the current video playback, the first audio segment is sent to the server, so that the server searches for audio data of the background music based on the first audio segment.

[0277] In some embodiments, the present application can continuously record audio clips of background music, and continuously send newly recorded audio clips to the server, so that the server generates a combined clip for song identification after splicing the first audio clip after receiving the song identification request and the subsequent cumulatively received audio clips. Until the server 400 feeds back the audio data of the background music to the display device. Continuously recording audio clips of background music is uninterrupted recording, and continuously sending the recorded audio clips to the server 400 is periodically uploading the audio clips to the server 400. The period for uploading the audio clips to the server 400 can be gradually shortened. For example, the first period can be set to a longer period (1s) in order to provide as many spectral features as possible, the second period can be 50ms, and the third period can be 25ms... This can not only ensure sufficient spectral features to ensure recognition results, but also improve recognition efficiency. Therefore, the first audio clip in the embodiment of the present application refers to the audio clip sent by the display device 200 to the server 400 in the first period.

[0278] Exemplarily, when the display device does not receive the recognition result, it can continue to record the second audio clip, and in response to receiving the recognition result, upload the second audio clip to the server, and the server splices the first audio clip and the second audio clip to generate a combined clip for song identification; when the display device does not receive the recognition result, it can continue to record the third audio clip, and in response to receiving the recognition result, upload the third audio clip to the server, and the server splices the first audio clip, the second audio clip and the third audio clip to generate a combined clip for song identification; .... In this way, the collection cycle of the audio clip can be shortened, and the same collection time as in the cumulative related art cannot be applied to guarantee the success of song identification. Moreover, for some scenarios (for example, when the collected sound clip has more obvious spectral characteristics), song identification can be achieved without collecting the collection time in the related art.

[0279] In certain embodiments, audio data can be the audio content identified according to the accumulated sound fragments received. The duration threshold of the server 400 feedback audio data is set. If it exceeds the duration threshold, a timeout prompt is popped up on the user interface to prompt the user to currently search for an audio data supermarket. If no relevant music is searched, a prompt message that the relevant audio is not searched can also be popped up on the user interface. If the music player software currently installed by the display device 200 does not have the playback copyright for the background music, a prompt message that the playback copyright is not available can be popped up on the user interface. The embodiment of the present application is illustrative of the scheme with the fact that the server 400 retrieves audio data, the feedback audio data does not time out, and the music player software currently installed by the display device 200 has the playback copyright for the background music.

[0280] The specific process of searching for audio data is as follows: Server 400 establishes a communication relationship with display device 200, receives the first audio clip of the background music currently being played by the video player, and searches the database based on the first audio clip of the background music currently being played by the video player to obtain the music information of the background music currently being played by the video player. Server 400 then feeds back the music information of the background music currently being played by the video player to display device 200. The database in server 400 may be pre-configured and may store audio data of various background music and the corresponding background music information. The database may also store the background music information expected to correspond to audio data processed by an audio fingerprint algorithm.

[0281] If the server 400 cannot match the corresponding audio data in the database after receiving the first audio clip, the server 400 continues to receive the second audio clip uploaded by the display device 200. After splicing the first audio clip and the second audio clip, the server 400 generates a new audio clip (for example, an audio clip with a new audio fingerprint obtained by processing with an audio fingerprint algorithm), and then searches the database for audio data matching the fingerprint based on the new audio clip. If the corresponding audio data still cannot be matched in the database, the server 400 continues to receive the third audio clip uploaded by the display device 200. After splicing the first audio clip, the second audio clip and the third audio clip, the server 400 generates a new audio clip again, and then searches the database for audio data matching the fingerprint based on the new audio clip. If the corresponding audio data is matched in the database, the server 400 feeds back the audio data to the display device 200.

[0282] The song recognition button in the embodiment of the present application can be a control UI that is directly displayed on the video player application interface, or it can be a hidden control UI. If it is a hidden control UI, the hidden control UI will pop up and display only when the user calls out the hidden song recognition button through the remote control. For example, the song recognition button is not displayed on the video player application interface at the beginning. The user needs to click on the video player application interface first, and the song recognition button is displayed on the video player application interface. In order to avoid the user from accidentally clicking on the song recognition button, after the song recognition button is displayed on the video player application interface after the above operation, the focus does not immediately fall on the song recognition button, but the user needs to operate the remote control to move the focus to the song recognition button before clicking the song recognition button.

[0283] The control UI can be the player's playback UI, which can include a video player application interface for accessing background music and a UI for displaying the searched background music information. The background music entry control UI can be displayed as a graphical control, and the searched background music information results UI can be a sidebar added to the entry control UI graphic to display the searched background music information.

[0284] There are two ways to implement the above-mentioned acquisition of the first audio clip of the background music while playing the background music on the same display device 200:

[0285] In a first implementation, after an audio output device (e.g., a speaker) of the display device 200 outputs background music, an audio receiving device (e.g., a microphone) of the display device 200 receives the background music, and then an audio decoder of a video player on the display device 200 decodes the received background music to obtain a first audio clip of the background music.

[0286] The second implementation is that before the audio output device (such as a speaker) of the display device 200 outputs the background music, the display device 200 has decoded and obtained the first audio segment of the background music (for example, before the display device 200 plays the TV series, it obtains all the background music included in the TV series in advance and stores the audio data of all the background music in the memory). In this implementation, the display device 200 does not need to use an audio receiving device to receive the background music and decode it again to obtain the first audio segment. It only needs to read the first audio segment at the current moment from the memory after receiving the song recognition instruction. Compared with the first implementation, the second implementation can avoid the distortion and redundancy of the content of the received first audio segment due to external interference or failure of the audio receiving device during the secondary reception of background music. The second implementation can achieve accurate background music information even in a noisy environment. In addition, the second implementation can achieve accurate background music information even without an audio output device or an audio receiving device.

[0287] If the first audio segment is obtained in the first implementation manner, the audio receiving device continuously receives the background music without recording any time point.

[0288] If the second implementation method is used to obtain the first audio clip, the specific process may be:

[0289] After the display device 200 receives the song identification instruction, it generates a search time period with the time point of receiving the song identification instruction as the starting time point. A specific method of generating the search time period can be to obtain the end time point by adding a predetermined time length to the starting time point. The display device 200 decodes the background music of the time period based on the starting time point and the end time point to obtain the first audio clip of the background music being played by the video player. The display device 200 then sends the obtained first audio clip to the server 400, so that the server 400 searches for background music based on the first audio clip.

[0290] The above solution can be implemented by the core processor of the video player on the display device 200. In addition to being responsible for decoding the background music being played, the video player core can also be responsible for functions such as video reading (for example, downloading video data from the network), audio and video decoding, and audio and video synchronization. When the user triggers a function that requires searching for background music information, and the video player core decodes and obtains the first audio clip of the current background music, the video player core is also responsible for additionally transmitting the decoded audio data to the audio fingerprint calculation and nursing device used for fingerprinting the audio data to calculate the audio fingerprint.

[0291] The detailed process of the display device 200 acquiring the background music information of the background music being played is as follows:

[0292] An audio fingerprint calculation and processing device uses an audio fingerprint algorithm to perform fingerprint processing on a first audio segment of background music being played by a video player to generate an audio fingerprint corresponding to the first audio segment.

[0293] The audio fingerprint here can be an acoustic fingerprint, which can be a digital summary extracted from the audio signal of the background music through a specific algorithm, representing the digital signature of the background music in terms of acoustic feature content.

[0294] The audio fingerprint information is then sent to a third-party server via a network transmission device. A query is performed on the audio fingerprint database on the third-party server to obtain the background music information of the background music currently being played by the player. The database is used to store an audio fingerprint data table, which records pre-set audio fingerprint information and its corresponding background music information.

[0295] In this application, the above-mentioned audio fingerprint is mainly used to filter out similar or identical audio from the audio database and further obtain the background music information corresponding to the audio data. Here, the audio database stores the audio fingerprint of at least one audio data and its corresponding background music information (for example, the song title, composer, lyrics, playback address, download address, and singer of the background music). The audio fingerprint can be used as the index of the corresponding background music information in the database. The audio fingerprint calculation module must use the audio fingerprint algorithm used by the audio fingerprint stored in the database.

[0296] It should be noted that in the embodiment of the present application, the current video playing may be background music, or it may be opening music, ending music, interlude, etc. The embodiment of the present application uses background music as an example to illustrate the solution. If the current video playing is other sound data such as opening music, ending music, interlude, etc., the method of the embodiment of the present application can also be used to identify songs.

[0297] In the embodiment of the present application, the first audio clip specifically represents a clip of background music, while the audio data represents audio content similar to or identical to the background music. That is, the audio data may or may not be the same version as the background music. The audio data and the background music may have a high similarity in waveform spectrum or high similarity in sound text. For example, if the background music is song A sung by Li Si, the audio data may be song A sung by Li Si or song A sung by Wang Wu.

[0298] In practical applications of the background music search method of this application, the relationship between the searched audio data and the sound data may exist in the following four situations:

[0299] In the principle diagram of the relationship between the sound data of the first audio clip and the audio data shown in Figure 23, the first case is that the sound data and the audio data are completely different (completely different here can refer to the completely different scope of the audio content), that is, there is no inclusion relationship or intersection relationship between the sound data and the audio data, then the server 400 feeds back the audio data to the display device 200, and the audio player on the display device 200 plays the audio data.

[0300] In the principle diagram of the relationship between sound data and audio data as shown in Figure 24, the second case is that if the audio data and the sound data are exactly the same, then the server 400 does not need to return the audio data to the display device 200, and the user can hear the complete version of the background music when watching the currently playing video. It is also possible that the currently existing resources are not complete versions, such as songs from a long time ago, so the complete version referred to in the embodiment of the present application refers to the most complete version that can be searched, rather than the original complete version of the background music. The user interface as shown in Figure 25 pops up the first prompt message in the user interface, prompting the user that the audio data currently found are identical to the sound data. At this time, since there is no need to open the audio player to play the audio data, there is no need to control the video player to pause the currently playing video.

[0301] In the third case, as shown in the principle diagram of the relationship between the sound data of the first audio segment and the audio data in FIG26 , the sound data and the audio data have an intersection. The server 400 then feeds the audio data back to the display device 200 , and the audio player on the display device 200 plays the audio data.

[0302] In the principle diagram of the relationship between sound data and audio data shown in Figure 27, the fourth case is that the audio data includes sound data, then the server 400 feeds the audio data back to the display device 200, and the audio player on the display device 200 plays the audio data.

[0303] The signaling diagram shown in Figure 28 includes the following steps:

[0304] S2801: The user sends a song identification instruction to the display device 200;

[0305] S2802: The display device 200 sends a first audio clip to the server 400;

[0306] S2803: The server 400 feeds back the audio data to the display device 200;

[0307] S2804: The audio player on the display device 200 plays the audio data while the display device 200 controls the video player to pause playing the current video, and records the first playing progress of the current video when the current video is paused.

[0308] S2805: The user sends an audio stop playback instruction to the display device 200;

[0309] S2806. If the audio data is played completely or the user clicks the stop play button on the audio player, the processor may receive an audio stop play instruction (at this time, the audio stop play instruction may be automatically triggered by the audio player or generated by receiving a click operation from the user). If the audio stop play instruction is automatically triggered by the audio player due to the completion of the audio data playback, the audio player automatically stops playing the audio data. If the audio stop play instruction is generated due to receiving a click operation from the user, the processor needs to control the audio player to stop playing the audio data according to the audio stop play instruction. After the audio player stops playing the audio data, it feeds back the playback result (i.e., stops playing the audio data) to the processor. The processor controls the video player to continue playing the current video based on the first playback progress according to the playback result.

[0310] For example, when a user is watching TV series XX through a video player (video progress is 25 minutes), background music YY is played. The user clicks the "Listen to Song and Identify Song" button in the video player page (video progress is 25 minutes and 10 seconds), and the display device 200 sends the first audio clip of background music YY (a clip of background music YY) to the server. The server 400 feeds back the audio data of background music YY (which can be the full version of background music YY) to the display device 200 at the 25th minute and 15 seconds of the video progress based on the first audio clip of background music YY. At the same time, the playback progress of TV series XX is paused at the 25th minute and 15 seconds of the video progress. The display device 200 starts to play the audio data of background music YY on the audio player. After the audio data of background music YY is played, the audio player is controlled to close, and the video player continues to play TV series XX from the 25th minute and 16 seconds of the video progress (at this time, TV series XX may still sound the clip of background music YY that has not been played, that is, the sound data has not been played).

[0311] In some embodiments, after receiving the audio data sent by the server 400, the display device 200 needs to decode the audio data. After the decoding is completed, the display device 200 pulls up the audio player, and then the display device 200 controls the video player to pause playing the current video.

[0312] In some embodiments, before the display device 200 controls the video player to pause playing the current video, if the user interrupts the song recognition operation, for example, the user presses the return button on the remote control or clicks the cancel control on the song recognition application, the display device 200 responds to the return instruction or the cancel instruction, and does not control the video player to pause playing the current video. At the same time, even if it receives audio data feedback from the server, it does not display anything related to song recognition on the page, but continues to play the current video.

[0313] The media resource playback method provided by the embodiment of the present application is that the processor obtains the first audio clip of the background music of the current video playback in response to the song identification instruction input by the user, wherein the first audio clip is a clip of the background music. And the display device sends the first audio clip to the server. The server searches for the audio data of the background music based on the first audio clip. The audio data is a clip or a complete version of the background music, and the audio data at least includes the first audio clip. If the audio data also includes other audio data, the audio data fed back by the server is received, and the processor controls the video player to pause the current video, records the first playback progress of the current video, and controls the audio player to play the audio data. In response to the audio stop playback instruction input by the user, the audio player is controlled to stop playing the audio data, and at the same time, the video player is controlled to continue playing the current video based on the first playback progress. In this way, if the user is interested in the background music and is willing to play other parts of the background music, the user only needs to input the song identification instruction to achieve the purpose of audio and video linkage, thereby improving the user experience.

[0314] In some embodiments, after the server feeds back the audio data, the server can wait for the current video to finish playing the current background music before pausing the current video. Furthermore, the position of the current video playing the partial segment of the current background music can be recorded. After the current video finishes playing the background music segment, the audio player can be controlled to continue playing the partial segment of the background music instead of playing the full version of the background music.

[0315] The specific implementation process of the audio-visual linkage of this embodiment can be shown in the signaling diagram of FIG29 , which specifically includes the following steps:

[0316] S2901. The user enters a song identification instruction by clicking the song identification button in the video playback application page, and sends the song identification instruction to the song identification application (the user can also directly click the song identification application, in which case the video player still plays the current video normally), where the song identification application can be an audio playback application with a song identification function.

[0317] S2902: The song recognition application determines the service APP currently in the playback state, such as the video playback application of this application. The song recognition application sends a communication establishment request to the video playback application. After the song recognition application and the video playback application establish the communication request, the song recognition application and the video playback application interact with each other.

[0318] S2903: The video player returns the playback progress information of the background music to the music recognition application (for example, the position of the specific audio clip of the background music currently being played in the video).

[0319] S2904. The music recognition application requests an audio file from the audio output module (the audio output module may be an audio player, and the audio file is the audio data of the background music being played, and may also include playback information of the audio data);

[0320] S2905. The audio output module returns the audio file to the song recognition application.

[0321] S2906. After receiving the audio file, the song recognition application continues to transmit the sound clips in the audio file to the song recognition cloud until it times out or returns the recognition result.

[0322] S2907. The song recognition cloud returns recognition information to the song recognition application.

[0323] S2908. After receiving the returned identification information, the song recognition application searches for the corresponding song in the song recognition application according to the identification information.

[0324] S2909. At the same time, the music recognition application notifies the video player to pause the video playback (at this time, the background music segment has been played) according to the playback progress information of the background music (that is, the current video has finished playing the background music segment).

[0325] S2910: After the video player stops playing the video, it feeds back a video progress interruption message to the song recognition application;

[0326] S2911. After receiving the video progress interruption information, the song recognition application starts to play the searched song according to the playback progress information of the background music (it can be a full version or it can be connected to the progress of the background music in the current video playback).

[0327] When the song recognition application is playing a searched song, if the user inputs an audio stop playback command, the song recognition application stops playing the searched song in response to the command. If the user inputs a command to exit the song recognition application at this time, the song recognition application is exited in response to the command, and the video player continues playing the current video according to the recorded video progress interruption information.

[0328] S2912. If the searched song is finished playing (not considering the situation where the song recognition application automatically jumps to play other songs after playing the searched song), the user does not need to enter an instruction to exit the song recognition application. The song recognition application automatically sends an audio stop playback instruction to the video player. After receiving the instruction, the video player continues to play the current video according to the video progress interruption information.

[0329] For example, when a user is watching TV series XX through a video player (the video progress is at the 25th minute), background music YY is played. The user clicks the "Listen to Song and Identify Song" button on the video player page (the video progress is at the 25th minute and 10 seconds), and the display device 200 sends the first audio clip of background music YY (a clip of background music YY) to the server. The server 400 feeds back the audio data of background music YY (which can be the complete version of background music YY) to the display device 200 based on the first audio clip of background music YY at the 25th minute and 15 seconds of the video progress. The display device 200 waits for the first audio clip of background music YY to be played after completion (the video progress is at the 25th minute and 30 seconds) before pausing the playback of TV series XX, that is, the playback progress of TV series XX is paused at the 25th minute and 30 seconds of the video progress. The display device 200 starts playing the audio data of background music YY on the audio player. After playing the audio data of background music YY, the audio player is controlled to close, and the video player continues to play TV series XX from the 25th minute and 31 seconds of the video progress (at this time, since the first audio clip of background music YY has been played, the background music YY clip will no longer be played when playing TV series XX).

[0330] It should be noted that the server used for song identification and the server used for song search in this application can be different servers or the same server. The song identification cloud given in the above embodiment is a server for song identification, and the server for song search can also be a song identification cloud. For example, third-party music software usually has song identification and song search functions, so the song identification server and the song search server can both be third-party music software servers. The song identification server can also be a local server, and the local server is preset with a database for song identification. For example, the database mentioned above stores an audio fingerprint data table, which is used to record pre-set audio fingerprint information and its corresponding background music information.

[0331] In some embodiments, if the audio content of the sound data does not correspond to the end of the audio data, audio content similar to or identical to the first audio segment in the audio data is intercepted, and the audio data is played starting from the end point of the intercepted audio content. If the first audio segment is the end of the audio data, the audio data may be played starting from the starting point of the audio data.

[0332] For example, when a user is watching TV series XX through a video player, background music YY is played. The user clicks the "Listen to Song and Identify Song" button on the video player page, and the display device 200 sends the first audio clip of background music YY to the server. The server 400 feeds back the audio data of background music YY (the complete version of the background music) to the display device based on the first audio clip of background music YY. The total length of background music YY is 3 minutes and 20 seconds. If the first audio clip is the last 20 seconds, the audio player starts playing background music YY from the starting point of the background music, that is, from 0:01. If the first audio clip is a clip from 1 minute 10 seconds to 1 minute 30 seconds, the audio player starts playing the audio data from 1 minute 31 seconds.

[0333] In some embodiments, the display device 200 further receives music information of the background music fed back by the server 400, and controls the display of the music information on the display 200. If the music information includes lyrics of the background music, the lyrics corresponding to the first audio clip are controlled to be highlighted.

[0334] For example, the display device 200 also receives music information of background music fed back by the server 400, which includes the song title, lyrics, background image, etc. When a user is watching TV series XX through a video player and background music YY plays, the user interface shown in FIG30 is ultimately displayed, in which the lyrics of the segment of background music YY used in TV series XX, "Tears Frozen on the Young Face," are magnified compared to the lyrics of other segments (in addition to magnification, highlighting, darkening, and other prominent display methods can also be adopted). This allows the user to distinguish between the segment used in the TV series and other segments, thereby enhancing the user's perception of the background music and improving the user experience.

[0335] In some embodiments, the character image in the current video screen is identified, and the music information is controlled to be displayed on the current video screen in a manner that does not block the character image. In the user interface shown in Figure 31, although the currently playing TV series XX is paused, the video screen is still displayed in the interface. At the same time, the music information (background picture, title, lyrics, etc.) of the background music YY is displayed below the video screen so that the music information does not block the character image display. Since background music is usually associated with the plot of the TV series, it has the effect of enhancing the plot. Therefore, the user can continue to watch the video screen while enjoying the background music, thereby further enhancing the user's perception of the background music and improving the user experience.

[0336] In certain embodiments, the condition that the audio player plays the audio data further comprises that the audio player supports the format of the audio data. If the audio player does not support the format of the audio data, a second prompt message is ejected on the display, wherein the second prompt message is used to prompt the user to install the audio player that supports playing the audio data.

[0337] In the user interface shown in Figure 32, after the display device 200 receives the full version of the background music YY fed back by the server 400, the audio player installed on the display device 200 does not support the full version of the background music YY, or no audio player is installed on the display device 200. Then a prompt message pops up on the user interface, prompting the user to install the corresponding audio player that supports playing the background music YY. A list of audio playback applications that can be installed can also be displayed on one side of the prompt message box that pops up. The audio playback application list is displayed by audio playback application 1, audio playback application 2, and audio playback application 3. The user can also click on the corresponding audio playback application control to directly install the application. For example, if the user clicks on the control of audio playback application 1, audio playback application 1 is directly installed on the display device 200. After the installation of audio playback application 1 is successful, background music YY is then played through audio playback application 1.

[0338] The present application provides a method for playing media assets. FIG33 is a flow chart illustrating a method for playing media assets according to an exemplary embodiment. The method for playing media assets is applicable to the display device 200 in the system shown in FIG21 . As shown in FIG33 , the method for playing media assets may include the following steps:

[0339] In step S101, in response to the received song identification instruction, the first audio segment of the sound data of the current video playback is collected, and a song identification request containing the first audio segment is sent to the server, so that the server searches for the audio data of the sound data based on the first audio segment, wherein the first audio segment represents a segment of the sound data, and the audio data represents audio content that is similar to or identical to the sound data.

[0340] In step S102, if the audio content of the audio data exceeds the audio content range of the sound data, the audio data fed back by the server is received, the video player is controlled to pause playing the current video, the first playback progress of the current video when the current video is paused is recorded, and the audio player is controlled to play the audio data.

[0341] In step S103, in response to the received audio stop playback instruction, the audio player is controlled to stop playing the audio data, and the video player is controlled to continue playing the current video based on the first playback progress.

[0342] In some embodiments, if the audio content of the audio data does not exceed the audio content range of the sound data, the video player is not controlled to pause playing the current video.

[0343] In some embodiments, if the audio content of the sound data does not correspond to the end portion of the audio data, the audio data is played starting from the end point of an audio content that is similar to or identical to the audio content of the sound data. If the audio content of the sound data does not correspond to the end portion of the audio data, the audio data is played starting from the starting point of the audio data.

[0344] In some embodiments, the display device further receives music information of the background music fed back by the server and controls the display of the music information on the display. The music information includes lyrics of the background music, and the lyrics corresponding to the sound data are controlled to be highlighted.

[0345] It will be appreciated by those skilled in the art that various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as a "data block," "processor," "engine," "unit," "component," or "system." In addition, various aspects of the present application may be represented by a computer product located in one or more computer-readable media, the product including computer-readable program code.

[0346] For ease of explanation, the above description has been made with reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations are possible. The above embodiments are selected and described to better explain the principles and practical applications, so that those skilled in the art can better utilize the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, comprising: a display configured to display an image and / or a user interface; a memory configured to store computer instructions and data associated with said display; At least one processor, connected to the display and the memory, configured to execute the computer instructions so that the display device: In response to an audio recognition request input by a user, establishing communication with a first application that is currently playing audio, and obtaining historical audio data played before a current playback moment from a media resource cache corresponding to the first application; By performing a recognition operation on the audio data to be recognized corresponding to the historical audio data, a recognition result of the audio data to be recognized is determined, and the recognition result is displayed in the audio recognition display area of ​​the display.

2. The display device according to claim 1, wherein the processor is further configured to execute computer instructions so that the display device: After obtaining the historical audio data played before the current playback moment from the media resource cache corresponding to the first application, obtaining the recording data played by the first application within a first preset time period after receiving the audio recognition request; In the step of performing a recognition operation on the audio data to be recognized corresponding to the historical audio data, the processor is configured to: By performing a recognition operation on the audio data to be recognized, wherein, The audio data to be identified includes the historical audio data, or the audio data to be identified includes the historical audio data and the recorded data.

3. The display device according to claim 2, wherein if the historical audio data includes first sub-segment data of a second segment data currently being played that was played before the current playback moment, in the step of obtaining the recorded data played by the first application within a first preset time period after receiving the audio recognition request, the processor is further configured to execute computer instructions so that the display device: Acquire first candidate segment data played by the first application within a second preset time after the current playing moment of the historical audio data, the recorded data including the first candidate segment data; or, comparing the duration of the first sub-segment data with a preset recognition duration; If the duration of the first sub-segment data is less than the preset recognition duration, determining a first playback duration, where the first playback duration is the difference between the preset recognition duration and the duration of the first sub-segment data; Acquire first candidate segment data played by the first application within the first playback duration after the current playback moment of the historical audio data, the recording data including the first candidate segment data.

4. The display device according to claim 2, wherein if the historical audio data includes, within a second segment data being played, first sub-segment data that was played before the current playback moment, and first segment data that was played before the second segment data, in the step of obtaining the recorded data played by the first application within a first preset time period after receiving the audio recognition request, the processor is further configured to execute computer instructions so that the display device: Acquire second candidate segment data played by the first application within a third preset time after the current playing moment of the historical audio data, the recorded data including the second candidate segment data; or, comparing a duration of the played data with a preset identification duration, where the duration of the played data is the sum of a duration of the first sub-segment data and a duration of the first segment data; If the duration of the played data is less than the preset identification duration, determining a second play duration, where the second play duration is the difference between the preset identification duration and the duration of the played data; Acquire second candidate segment data played by the first application within the second playback duration after the current playback moment of the historical audio data, the recording data including the second candidate segment data.

5. The display device according to claim 1, if the historical audio data includes first sub-segment data of the second segment data being played that was played before the current playback time, or the historical audio data includes first segment data and the first sub-segment data that were played before the second segment data; and the media asset cache also includes tail audio data that was downloaded and not played after the current playback time, the processor is further configured to execute computer instructions so that the display device: Obtaining the last audio data; In the step of performing a recognition operation on the audio data to be recognized corresponding to the historical audio data, the processor is further configured to execute computer instructions to enable the display device to: By performing a recognition operation on the audio data to be recognized, wherein, The audio data to be recognized includes the historical audio data and the end audio data.

6. The display device according to claim 5, if the end audio data includes second sub-segment data in the second segment data that has been downloaded and not played after the current playback moment, in the step of performing an identification operation on the audio data to be identified, the audio data to be identified includes the historical audio data and the second sub-segment data.

7. The display device according to claim 5, if the end audio data includes the second sub-segment data in the second segment data that has been downloaded and not played after the current playback moment and the third segment data that has been downloaded and not played after the second segment data, in the step of performing the recognition operation on the audio data to be recognized, the audio data to be recognized includes the historical audio data, the second sub-segment data and the third segment data.

8. The display device according to claim 1 , wherein in the step of determining a recognition result of the audio data to be recognized by performing a recognition operation on the audio data to be recognized corresponding to the historical audio data, the processor is further configured to execute computer instructions so that the display device: Sending the audio data to be recognized to a recognition audio server; Receive the recognition result of the audio data to be recognized returned by the recognition audio server.

9. The display device according to claim 1 , wherein the processor is further configured to execute computer instructions so that the display device: In response to a user's request to identify audio, obtaining initial audio data of a preset first duration, and sending the initial audio data and data information of the initial audio data to a server, the data information including an audio coding identifier and a first timing identifier; Acquire intermediate audio data of a preset second duration, and send the intermediate audio data to the server, where the intermediate audio data includes the audio coding identifier and the second timing identifier; receiving information identifying target audio data from the server, and displaying the information on the display, wherein the target audio data includes the initial audio data, or the initial audio data and the intermediate audio data; in, The preset first duration is greater than or equal to the preset second duration, and the sum of the preset first duration and the preset second duration is less than the recording start and end durations corresponding to the request.

10. The display device according to claim 9, wherein The first timing identifier is determined based on the time when the initial audio data is acquired, and the second timing identifier is determined based on the time when the intermediate audio data is acquired.

11. The display device according to claim 9, wherein In the step of obtaining initial audio data based on a preset first duration in response to a user's request to recognize audio, the processor is further configured to execute computer instructions to cause the display device to: In response to a user's request to identify audio, recording first audio data based on the preset first duration; Encoding the first audio data using a preset encoding and decoding strategy to determine the initial audio data; In the step of acquiring the intermediate audio data based on the preset second duration, the processor is further configured to execute computer instructions to enable the display device to: Recording second audio data based on the preset second duration; The second audio data is encoded using a preset encoding and decoding strategy to determine the intermediate audio data.

12. The display device according to claim 1, further comprising: Audio player, used to play audio data; Video player, used to play video data and audio data; A user input interface for receiving input signals; The processor is further connected to the user interface, and the processor is further configured to execute computer instructions to cause the display device to: In response to the received song identification instruction, collecting a first audio segment of the sound data of the current video playback, and sending a song identification request including the first audio segment to a server, so that the server searches for audio data of the sound data according to the first audio segment, wherein the first audio segment represents a segment of the sound data, and the audio data represents audio content similar to or identical to the sound data; If the audio content of the audio data exceeds the audio content range of the sound data, receiving the audio data fed back by the server, controlling the video player to pause playing the current video, recording the first playing progress of the current video when the current video is paused, and controlling the audio player to play the audio data; In response to the received audio stop playback instruction, the audio player is controlled to stop playing the audio data, and the video player is controlled to continue playing the current video based on the first playback progress.

13. The display device according to claim 12, wherein: The processor is further configured to execute computer instructions to cause the display device to: If the audio content of the audio data does not exceed the first audio segment of the audio content range of the sound data, the video player is not controlled to pause playing the current video.

14. The display device according to claim 12, wherein: If the audio content of the sound data does not correspond to the end of the audio data, the processor is further configured to execute computer instructions to enable the display device to: The audio content similar to or identical to the first audio segment in the audio data is intercepted, and the audio data is played starting from the end point of the intercepted audio content.

15. The display device according to claim 12, wherein: If the audio content of the sound data does not correspond to the end of the audio data, the processor is further configured to execute computer instructions to enable the display device to: The audio data is played starting from a starting point of the audio data.

16. The display device according to claim 13, wherein The processor is further configured to execute computer instructions to cause the display device to: Receive the music information of the background music fed back by the server, and control the display of the music information on the display.

17. The display device according to claim 16, wherein: The music information includes lyrics of the background music, and the processor is further configured to execute computer instructions to enable the display device to: Controlling the highlighting of lyrics corresponding to the sound data.

18. The display device according to claim 12, wherein: The condition for the audio player to play the audio data further includes that the audio player supports the format of the audio data; If the audio player does not support the format of the audio data, a second prompt message is popped up on the display, wherein the second prompt message is used to prompt the user to install an audio player that supports playing the audio data.

19. An audio recognition method, comprising: In response to an audio recognition request input by a user, establishing communication with a first application that is currently playing audio, and obtaining historical audio data played before a current playback moment from a media resource cache corresponding to the first application; By performing a recognition operation on the audio data to be recognized corresponding to the historical audio data, a recognition result of the audio data to be recognized is determined, and the recognition result is displayed in an audio recognition display area of ​​the display.

20. The audio recognition method according to claim 19, after obtaining historical audio data played before the current playback moment from the media resource cache corresponding to the first application, further comprising: Acquire recorded data played by the first application within a first preset time period after receiving the audio recognition request; In the step of performing a recognition operation on the audio data to be recognized corresponding to the historical audio data, the audio data to be recognized includes the historical audio data, or the audio data to be recognized includes the historical audio data and the recorded data.

Images