Audio recommendation method and device, storage medium and electronic equipment

Abstract

The invention discloses an audio recommendation method and device, a storage medium and electronic equipment. The method comprises the steps that at least one piece of media source information is acquired, and the media source information is media information containing at least one piece of non-audio data; audio applicable attributes associated with each piece of media source information in the at least one piece of media source information are obtained, and the audio applicable attributes are used for representing the applicability of the media source information when the media source information is played as audio; and determining recommended media source information suitable for being played as an audio from the at least one piece of media source information by using the audio application attribute, and carrying out recommended playing on the audio corresponding to the recommended media source information. The method can be applied to scenes such as cloud technology and artificial intelligence. The technical problem that the audio recommendation efficiency is low is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to an audio recommendation method, device, storage medium and electronic device. Background Art

[0002] Existing audio recommendation technologies often use professionals to create and input corresponding audio data into the playback platform, such as reading audio. However, this method requires professional audio creators to spend a lot of time on audio creation, which leads to a small number of audio recommendations and high time cost, thus causing technical problems of low efficiency of audio recommendation.

[0003] Therefore, there is a technical problem of low efficiency of audio recommendation in the related art. Summary of the invention

[0004] The embodiments of the present application provide an audio recommendation method, device, storage medium and electronic device to at least solve the technical problem of low efficiency of audio recommendation in the related art.

[0005] According to one aspect of an embodiment of the present application, there is provided an audio recommendation method, comprising: obtaining at least one media source information, wherein the media source information is media information containing at least one non-audio data; obtaining audio applicability attributes associated with each media source information in the at least one media source information, wherein the audio applicability attributes are used to indicate the applicability of the media source information when played as audio; using the audio applicability attributes, determining recommended media source information suitable for playing as audio from the at least one media source information, and recommending the audio corresponding to the recommended media source information for playback.

[0006] According to another aspect of an embodiment of the present application, an audio recommendation device is further provided, including: a first acquisition unit, used to acquire at least one media source information, wherein the above media source information is media information containing at least one non-audio data; a second acquisition unit, used to acquire audio applicability attributes associated with each media source information in the above at least one media source information, wherein the above audio applicability attributes are used to indicate the applicability of the above media source information when played as audio; a determination unit, used to determine, from the above at least one media source information, recommended media source information suitable for playing as audio by using the above audio applicability attributes, and recommend playing the audio corresponding to the above recommended media source information.

[0007] As an optional solution, the above-mentioned first acquisition unit includes: a first acquisition module, used to acquire at least one first media information, wherein the above-mentioned first media information is media information containing image data and audio data, the above-mentioned non-audio data includes the above-mentioned image data, and the above-mentioned media source information includes the above-mentioned first media information; the above-mentioned second acquisition unit includes: a second acquisition module, used to acquire the first applicability attribute associated with each first media information in the above-mentioned at least one first media information, wherein the above-mentioned first applicability attribute is used to indicate the applicability of the above-mentioned first media information when played as audio.

[0008] As an optional scheme, the above-mentioned second acquisition module includes: an identification submodule, which is used to perform voice recognition processing on the above-mentioned at least one first media information respectively to obtain the first text data corresponding to the above-mentioned each first media information, wherein the above-mentioned first text data is the text data after the above-mentioned first media information is converted into text, and the above-mentioned first applicable attribute includes the above-mentioned first text data; the above-mentioned determination unit includes: a first determination module, which is used to use the above-mentioned first text data to determine, from the above-mentioned at least one first media information, the first recommended media information suitable for playing as audio, and recommend the audio corresponding to the above-mentioned first recommended media information for playback, wherein the above-mentioned recommended media source information includes the above-mentioned first recommended media information.

[0009] As an optional scheme, the above-mentioned first determination module is used to include: a first determination submodule, which is used to determine, when the first text data contains first text density data and second text density data, from the above-mentioned at least one first media information, the above-mentioned first recommended media information whose first text density data is greater than the first preset threshold and the above-mentioned second text density data is greater than the second preset threshold, using the above-mentioned first text density data and the above-mentioned second text density data, wherein the above-mentioned first text density data is used to represent the density of the first duration distributed on the second duration, the above-mentioned second text density data is used to represent the density of the text data after the above-mentioned first media information is converted into text on the above-mentioned first duration, the above-mentioned first duration is used to represent the duration occupied by the above-mentioned first text data in the corresponding each of the above-mentioned first media information, and the above-mentioned second duration is used to represent the duration of the above-mentioned each of the above-mentioned first media information corresponding to the above-mentioned first text data.

[0010] As an optional solution, the first determination module includes: a first screening submodule, used to use the first text data to perform a primary screening on the at least one first media information to obtain at least one second media information, wherein the at least one first media information includes the at least one second media information; a determination submodule, used to input each second media information in the at least one second media information into an audible determination model for calculation to obtain a determination result of each second media information; a second screening submodule, used to use the determination result to perform a secondary screening on the at least one second media information to obtain the recommended media source information, wherein the recommended media source information is the second media information indicated as audible by the determination result, and the audible determination model is a neural network model pre-trained based on audible positive sample data and inaudible negative sample data, and is used to determine whether the input second media information is audible.

[0011] As an optional scheme, the above-mentioned determination submodule includes: an input subunit, which is used to input the audio input signal, text input signal and video cover image information of the above-mentioned each second media information into the above-mentioned audible discrimination model; a transformation subunit, which is used to perform spectrum transformation processing on the above-mentioned audio input signal to obtain a first input vector, perform text transformation processing on the above-mentioned text input signal to obtain a second input vector, and perform image transformation processing on the above-mentioned video cover image information to obtain a third input vector; a splicing subunit, which is used to perform feature splicing processing on the above-mentioned first input vector after feature extraction, the above-mentioned second input vector after feature extraction, and the above-mentioned third input vector after feature extraction to obtain media content vectors of the above-mentioned each second media information, wherein the above-mentioned audible discrimination model performs classification and determination processing according to the above-mentioned media content vectors to obtain the above-mentioned determination results of the above-mentioned each second media information.

[0012] As an optional scheme, the above-mentioned first acquisition unit includes: a third acquisition module, used to acquire at least one third media information, wherein the above-mentioned third media information is media information containing text data, the above-mentioned non-audio data includes the above-mentioned text data, and the above-mentioned media source information includes the above-mentioned third media information; the above-mentioned second acquisition unit includes: a fourth acquisition module, used to acquire the second applicability attribute associated with each third media information in the above-mentioned at least one third media information, wherein the above-mentioned second applicability attribute is used to indicate the applicability of the above-mentioned third media information when played as audio.

[0013] As an optional scheme, the above-mentioned fourth acquisition module includes: an acquisition sub-module, used to obtain the author category information, text category information and text length information associated with the above-mentioned each third media information, wherein the above-mentioned second applicable attribute includes the above-mentioned author category information, the above-mentioned text category information and the above-mentioned text length information; the above-mentioned determination unit includes: a second determination module, used to use the above-mentioned author category information, the above-mentioned text category information and the above-mentioned text length information to determine, from the above-mentioned at least one third media information, the second recommended media information suitable for playing as audio, and recommend the audio corresponding to the above-mentioned second recommended media information for playback, wherein the above-mentioned recommended media source information includes the above-mentioned second recommended media information.

[0014] As an optional scheme, the above-mentioned second determination module includes: a second determination submodule, which is used to determine fourth media information of a target author category, fifth media information of a target text category, and sixth media information of a target text length from the above-mentioned at least one third media information, wherein the above-mentioned target author category is the author category of the historical third media information that the user has read within a historical time period, the above-mentioned target text category is the text category of the above-mentioned historical third media information, the above-mentioned target text length is the text length of the above-mentioned historical third media information, and the above-mentioned at least one third media information includes the above-mentioned fourth media information, the above-mentioned fifth media information and the above-mentioned sixth media information; a third determination submodule, which is used to perform union processing on the above-mentioned fourth media information, the above-mentioned fifth media information and the above-mentioned sixth media information to obtain the above-mentioned second recommended media information.

[0015] As an optional solution, the above-mentioned device also includes: a detection module, which is used to perform security detection on data involved in playing each of the above-mentioned media source information as audio before obtaining the audio applicability attributes associated with each of the above-mentioned at least one media source information, wherein the media source information that passes the above-mentioned security detection is allowed to be played as audio for the above-mentioned recommendation.

[0016] As an optional solution, the above-mentioned device also includes: a fifth acquisition module, used to obtain whitelist media information from a whitelist media database, wherein the applicability of the media information in the above-mentioned whitelist media database when played as audio is set to be higher than a preset threshold, and the above-mentioned preset threshold is the minimum threshold for the above-mentioned recommended playback; a recommended playback module, used to perform the above-mentioned recommended playback of the audio corresponding to the above-mentioned whitelist media information.

[0017] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above audio recommendation method.

[0018] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the audio recommendation method through the computer program.

[0019] In an embodiment of the present application, at least one media source information is obtained, wherein the media source information is media information containing at least one non-audio data; the audio applicability attribute associated with each media source information in the at least one media source information is obtained, wherein the audio applicability attribute is used to indicate the applicability of the media source information when it is played as audio; using the audio applicability attribute, the recommended media source information suitable for playing as audio is determined from the at least one media source information, and the audio corresponding to the recommended media source information is recommended for playing. By screening from a large amount of media information containing non-audio data, the recommended media information suitable for playing as audio is obtained, and then the audio corresponding to the recommended media information is recommended for playing, and then there is no need to use professionals to create new audio, but audio screening and conversion are performed based on the existing massive materials of other types of media information (such as video), thereby achieving the purpose of reducing the time cost required for audio recommendation and increasing the number and type of audio recommendations, thereby achieving the technical effect of improving the efficiency of audio recommendation and solving the technical problem of low efficiency of audio recommendation in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a schematic diagram of an application environment of an optional audio recommendation method according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a process of an optional audio recommendation method according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an optional audio recommendation method according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an optional audio recommendation method according to an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of an optional audio recommendation device according to an embodiment of the present application;

[0026] Figure 6 A schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] According to one aspect of the embodiments of the present application, an audio recommendation method is provided. Optionally, as an optional implementation, the audio recommendation method can be but is not limited to being applied to Figure 1 In the environment shown, the environment may include, but is not limited to, a client 102 and a server 112 . The client 102 may include, but is not limited to, a display 104 , a processor 106 , and a memory 108 . The server 112 includes a database 114 and a processing engine 116 .

[0030] The specific process can be as follows:

[0031] Step S102, the client 102 obtains an initial time series prediction model 1001 and a plurality of time series samples 1002, wherein the plurality of time series samples 1002 are original time domain sample sets obtained by continuously sampling the original time series signal, and the time series prediction model 1001 is used to predict unknown time series data based on known time series data;

[0032] Steps S104-S106, the client 102 sends an audio recommendation request to the server 112, wherein the audio recommendation request is used to request to obtain at least one audio to be played;

[0033] Step S108, the server 112 responds to the audio recommendation request and obtains at least one piece of media source information from the database 114 through the processing engine 116, wherein the media source information is media information including at least one piece of non-audio data;

[0034] Step S110, the server 112 obtains the audio applicability attribute associated with each piece of media source information in the at least one piece of media source information, and determines the recommended media source information suitable for playing as audio from the at least one piece of media source information by using the audio applicability attribute;

[0035] Step S112, the server 112 determines the audio corresponding to the recommended media source information as at least one audio to be played in response to the audio recommendation request;

[0036] Steps S114-S116, sending the at least one audio to be played to the client 102 via the network 110, wherein the processor 106 in the client 102 is used to receive the at least one audio to be played, and process related audio information, and display the related audio information of the at least one audio to be played on the display 104, and store the at least one audio to be played and its related audio information in the memory 108.

[0037] remove Figure 1 In addition to the examples shown, the above steps can be completed independently by the client or the server, or by the client and the server working together, such as the client 102 executing the above steps S108 to S112, thereby reducing the processing pressure of the server 112. The client 102 includes but is not limited to a laptop, a tablet computer, a desktop computer, a smart TV, etc., and the present application does not limit the specific implementation of the client 102. The server 112 can be a single server or a server cluster consisting of multiple servers, or a cloud server.

[0038] Optionally, as an optional implementation, as Figure 2 As shown, the audio recommendation method can be executed by an electronic device, such as Figure 1 The client or server shown in the figure includes:

[0039] S202, obtaining at least one piece of media source information, wherein the media source information is media information including at least one piece of non-audio data;

[0040] S204, obtaining an audio applicability attribute associated with each piece of media source information in at least one piece of media source information, wherein the audio applicability attribute is used to indicate the applicability of the media source information when it is played as audio;

[0041] S206: Determine recommended media source information suitable for playing as audio from at least one piece of media source information by using the audio applicability attribute, and recommend playing the audio corresponding to the recommended media source information.

[0042] Optionally, in this embodiment, the above audio recommendation method can be applied to, but is not limited to, audio recommendation / audio creation scenarios. In this scenario, people often select audio that they are interested in and that is recommended by the platform for listening on a related audio push platform, wherein the audio recommended by the platform is often obtained from the audio created by the creator.

[0043] However, the above method requires professionals to create and enter the corresponding audio data into the playback platform, and requires professional audio creators to spend more time on audio creation, which leads to a small number of audio recommendations and high time cost, thus causing the problem of low efficiency of audio recommendations.

[0044] In order to solve the above-mentioned problem, the audio recommendation method provided in this embodiment is used to obtain recommended media information suitable for playing as audio by screening from massive media information containing non-audio data, and then recommend the audio corresponding to the recommended media information for playing. There is no need to employ professionals to create new audio, but audio screening and conversion are performed based on massive materials of other types of media information (such as video), thereby achieving the purpose of reducing the time cost required for audio recommendation and increasing the number and types of audio recommendations, thereby achieving the technical effect of improving the efficiency of audio recommendation and solving the problem of low efficiency of audio recommendation mentioned above.

[0045] Optionally, in this embodiment, the media source information may be, but is not limited to, media information including at least one non-audio data, wherein the non-audio data may be, but is not limited to, image data and text data.

[0046] To further illustrate, the media source information may be, but is not limited to, media information containing image data and audio data, such as video; the media source information may also be, but is not limited to, media information containing text data, such as public account articles.

[0047] Optionally, in this embodiment, the audio applicability attribute may be but is not limited to being used to indicate the applicability of the media source information when played as audio, wherein the applicability is used to indicate whether the audio after the media information is converted or extracted is suitable for playback, and the degree of suitability for playback.

[0048] To further illustrate, taking a video whose media source information includes image data and audio data as an example, the audio adaptation attribute associated with the video can be, but is not limited to, the proportion attribute of the text data corresponding to the audio data contained in the video distributed in the entire video, and the density attribute distributed in the audio data. Wherein, without considering the influence of other factors (such as the validity and compliance of the text), the higher the above-mentioned proportion attribute and the higher the above-mentioned density attribute, the richer the text content information contained in the video, and thus the better the suitability of the video when played as audio.

[0049] In addition, taking a public account article whose media source information contains text data as an example, the audio applicability attributes associated with the public account article may be but are not limited to the author category information, text category information and text length information associated with the public account article. Among them, the more the author category information, text category information and text length information conform to the preset user reading or listening habits, the better the applicability of the public account article when played as audio.

[0050] It should be noted that the above embodiment is only one of the optional methods, and new specific attributes and features for determining the applicability of media source information when played as audio may also be adjusted or set according to, but not limited to, actual application scenarios.

[0051] Optionally, in this embodiment, audio applicability attributes are utilized to determine recommended media source information suitable for playing as audio from at least one media source information, which may include, but is not limited to, including determining top N media source information with the best audio applicability attributes from at least one media source information, and may also include, but is not limited to, including determining media source information having audio applicability attributes higher than a preset attribute threshold from at least one media source information.

[0052] To further illustrate, taking the media source information as a video containing image data and audio data, when the audio adaptation attribute associated with the video is the proportion attribute of the text data corresponding to the audio data contained in the video distributed in the entire video, and the density attribute distributed in the audio data, the video with a proportion attribute higher than the proportion threshold (such as 50%) and a density attribute higher than the density threshold (such as 5 words / second) is determined as the recommended media source information.

[0053] In addition, taking the example of a public account article whose media source information is a text-type data, when the audio applicable attributes associated with the public account article may be but are not limited to the author category information, text category information and text length information associated with the public account article, the public account article whose number of attributes among at least three attributes such as the author category information, text category information and text length information that conform to the preset user reading or listening habits is greater than a preset number (such as 2) is determined as recommended media source information.

[0054] It should be noted that, when the recommended media source information is determined, the audio corresponding to the recommended media source information is recommended for playback.

[0055] To further illustrate, when the recommended media source information is the above-mentioned video, you can, but are not limited to, use corresponding audio extraction tools to extract audio from the above-mentioned video to obtain the corresponding audio; and, when the recommended media source information is the above-mentioned public account article, you can, but are not limited to, use corresponding text-to-speech technology, such as Text-to-Speech technology (TTS for short), to convert the above-mentioned public account article to obtain the corresponding audio.

[0056] Through the embodiments provided by the present application, recommended media information suitable for playing as audio is obtained by screening from massive amounts of media information containing non-audio data, and then the audio corresponding to the recommended media information is recommended for playback. There is no need to employ professionals to create new audio, but audio screening and conversion are performed based on massive materials of other types of media information (such as video), thereby achieving the purpose of reducing the time cost required for audio recommendation and increasing the number and types of audio recommendations, thereby achieving the technical effect of improving the efficiency of audio recommendation and solving the problem of low efficiency of audio recommendation mentioned above.

[0057] As an optional solution, obtaining at least one media source information includes:

[0058] S1, obtaining at least one first media information, wherein the first media information is media information including image data and audio data, the non-audio data includes image data, and the media source information includes the first media information;

[0059] Obtaining audio applicable attributes associated with each piece of media source information in at least one piece of media source information, including:

[0060] S2: Obtain a first applicability attribute associated with each first media information in at least one first media information, wherein the first applicability attribute is used to indicate the applicability of the first media information when being played as audio.

[0061] Optionally, in this embodiment, the first media information may be, but is not limited to, media information including image data and audio data, such as video, wherein at least one media source information may be, but is not limited to, including the first media information.

[0062] Optionally, in this embodiment, when at least one media source information includes first media information, a first applicability attribute associated with the first media information is obtained, wherein the audio applicability attributes associated with each media source information in the at least one media source information include the first applicability attribute associated with the first media information, and the first applicability attribute is used to indicate the applicability of the first media information when played as audio.

[0063] As an optional solution, obtaining the first applicable attribute associated with each first media information in at least one first media information includes:

[0064] S1, performing speech recognition processing on at least one first media information respectively to obtain first text data corresponding to each first media information, wherein the first text data is text data after the first media information is converted into text, and the first applicable attribute includes the first text data;

[0065] Determining recommended media source information suitable for playing as audio from at least one piece of media source information by using the audio applicability attribute, and recommending the audio corresponding to the recommended media source information for playing, including:

[0066] S2, using the first text data, determining first recommended media information suitable for playing as audio from at least one first media information, and recommending the audio corresponding to the first recommended media information for playing, wherein the recommended media source information includes the first recommended media information.

[0067] Optionally, in this embodiment, speech recognition processing is performed on the first media information to obtain first text data corresponding to the first media information, wherein the speech recognition processing can be but is not limited to ASR technology (Automatic Speech Recognition, referred to as ASR), so as to achieve the effect of recognizing the text result of the sound in the first media information.

[0068] Optionally, in this embodiment, the first text data may be, but is not limited to, used to indicate a proportion attribute of text data corresponding to the audio data included in the first media information distributed in the entire video, and a density attribute distributed in the audio data.

[0069] Optionally, in this embodiment, the first text data is used to determine, from at least one first media information, first media information whose proportion attribute and the density attribute both meet preset conditions, and then obtain first recommended media information suitable for playing as audio, wherein the preset conditions may be but are not limited to indicating that the proportion attribute is higher than a proportion threshold (such as 50%) and the density attribute is higher than a density threshold (such as 5 words / second).

[0070] As an optional solution, the first recommended media information suitable for playing as audio is determined from at least one first media information using the first text data, including:

[0071] S1. In the case where the first text data includes the first text density data and the second text density data, the first text density data and the second text density data are used to determine, from at least one first media information, the first recommended media information in which the first text density data is greater than the first preset threshold and the second text density data is greater than the second preset threshold, wherein the first text density data is used to represent the density of the first duration distributed over the second duration, the second text density data is used to represent the density of the text data after the first media information is converted into text over the first duration, the first duration is used to represent the duration occupied by the first text data in each corresponding first media information, and the second duration is used to represent the duration of each first media information corresponding to the first text data.

[0072] Optionally, in this embodiment, the first text density may include but is not limited to first text density data and second text density data, wherein the first text density data may be but is not limited to being used to represent the proportion of a first duration containing text data in the first media information to the total duration of the first media information (i.e., the second duration), and the second text density data may be but is not limited to being used to represent the average text density contained in the first media information within the first duration.

[0073] Optionally, in this embodiment, the first media information whose first text density data is greater than a first preset threshold and whose second text density data is greater than a second preset threshold in at least one first media information is determined as the first recommended media information.

[0074] To further illustrate, taking the first media information as a video including image data and audio data, the text result of the sound in the video is recognized by ASR technology, and the number of ASR texts in the video is M, the duration of ASR texts is M1, and the total duration of the video is M2. Then the first text density of the video is M1 / M2, and the second text density of the video is M / M1.

[0075] Further, when the first preset threshold is 0.5 and the second preset threshold is 5 words / second, the video with the first text density M1 / M2 greater than 0.5 and the second text density M / M1 greater than 5 words / second is determined as the first recommended media information.

[0076] It should be noted that the first recommended media information is screened and determined through the above-mentioned first text density data and second text density data to ensure that the first recommended media information contains enough audible content. At the same time, it is ensured that the audio content in the first recommended media information is of sufficient duration and has sufficient text density to ensure that there is enough audible content.

[0077] It should be noted that, based on the above-mentioned first text density data and second text density data, other optional attribute data included in the first text data may be added to further screen and determine the first recommended media information, but is not limited to the above-mentioned first text density data and second text density data.

[0078] To further illustrate, in the case where the first text density includes first text density data, second text density data, third text density data and fourth text density data, first recommended media information is determined from at least one first media information, wherein the first text density data is greater than a first preset threshold, the second text density data is greater than a second preset threshold, the third text density data is greater than a third preset threshold, and the fourth text density data is greater than a fourth preset threshold, wherein the third text density data is used to represent the number of title characters of the first media information, and the fourth text density data is used to represent the number of description characters of the first media information, and the third preset threshold and the fourth preset threshold can be but are not limited to 5 characters.

[0079] As an optional solution, the first recommended media information suitable for playing as audio is determined from at least one first media information using the first text data, including:

[0080] S1, using first text data, screening at least one first media information to obtain at least one second media information, wherein the at least one first media information includes at least one second media information;

[0081] S2, inputting each second media information of the at least one second media information into the audible determination model for calculation, and obtaining a determination result of each second media information;

[0082] S3, using the determination result, performing secondary screening on at least one second media information to obtain recommended media source information, wherein the recommended media source information is second media information indicated as audible by the determination result, and the audible discrimination model is a neural network model pre-trained based on audible positive sample data and inaudible negative sample data, and is used to determine whether the input second media information is audible.

[0083] Optionally, in this embodiment, the first text data is used to screen at least one first media information to obtain at least one second media information, which may include but is not limited to: using the above-mentioned first text density data and the above-mentioned second text density data to determine the first recommended media information whose first text density data is greater than a first preset threshold and whose second text density data is greater than a second preset threshold, and determining the first recommended media information as the above-mentioned at least one second media information.

[0084] It should be noted that the second media information obtained after the above screening is the media information that has enough audible content, sufficient audio content duration and sufficient text density in the first media information. It is understandable that the content is audible enough and the text density is high does not mean that the media information is necessarily suitable for listening. For example, even if the text density of the media information is high, it may be full of repeated text information, and even contain non-compliant content, such as uncivilized language.

[0085] Based on the above considerations, in this embodiment, after obtaining the at least one second media information based on the above text dimension through a primary screening, the pre-trained audible discrimination model is further used to calculate the judgment result, and the at least one second media information is secondary screened on the content dimension based on the judgment result to obtain recommended media source information with rich content that is audible enough and compliant, suitable and audible.

[0086] Optionally, in this embodiment, the audible discrimination model may be, but is not limited to, a neural network model pre-trained based on audible positive sample data and inaudible negative sample data, and is used to determine whether the input second media information is audible (ie, whether the content is appropriate).

[0087] As an optional solution, each second media information is input into the audible determination model for calculation to obtain a determination result of each second media information, including:

[0088] S1, inputting the audio input signal, text input signal and video cover image information of each second media information into an audible discrimination model;

[0089] S2, performing spectrum transformation processing on the audio input signal to obtain a first input vector, performing text transformation processing on the text input signal to obtain a second input vector, and performing image transformation processing on the video cover image information to obtain a third input vector;

[0090] S3, performing feature splicing processing on the first input vector after feature extraction, the second input vector after feature extraction, and the third input vector after feature extraction to obtain media content vectors of each second media information, wherein the audible discrimination model performs classification and judgment processing based on the media content vectors to obtain judgment results of each second media information.

[0091] Optionally, in this embodiment, in the case where at least one second media information is obtained after the above-mentioned screening, taking the current second media information included in the at least one second media information as an example, the audio input signal, text input signal and video cover image information of the current second media information are obtained, wherein the audio input signal may be but is not limited to the audio data included in the current second media information, the text input signal may be but is not limited to the text information obtained by performing speech recognition processing on the above-mentioned audio data, and the video cover image information may be but is not limited to the cover frame of the current second media information.

[0092] It should be noted that in this embodiment, taking the second media information as a video as an example, the internal frames of the video have little effect on whether it is audible, but the cover frame of the video, as the cover frame of the subsequent audio recommendation, has a certain influence on the listening decision, so it is used as one of the model inputs. The text information expressed by the audio information is very critical to whether the video is audible, so the text information obtained after speech recognition processing is used as one of the model inputs.

[0093] It can be understood that, in this embodiment, a multimodal neural network including visual information, auditory information and text information is constructed in combination with the audio signal to determine whether the second media information can be converted into audio suitable for listening.

[0094] Optionally, in this embodiment, the audio input signal is subjected to spectrum transformation processing to obtain the first input vector, which may be, but is not limited to, performing Logmel spectrum transformation processing on the audio input signal to extract audio features and describe frequency distribution characteristics of the audio to obtain the above-mentioned first input vector.

[0095] Optionally, in this embodiment, the text input signal is subjected to text transformation processing to obtain the second input vector, which may be, but is not limited to, embedding the text input signal through word / character vectors to obtain the corresponding second input vector.

[0096] Optionally, in this embodiment, the video cover image information is subjected to image transformation processing to obtain a third input vector, which may be used, but is not limited to, to indicate converting the video cover image information into an RGB three-channel image intensity expression vector to obtain the above-mentioned third input vector.

[0097] Optionally, in this embodiment, after obtaining the first input vector, the first input vector is input into a bidirectional GRU recurrent neural network for feature extraction; after obtaining the second input vector, the second input vector is input into multiple Transformer structures for feature extraction; after obtaining the third input vector, the third input vector is input into multiple CNN convolutional neural networks for feature extraction.

[0098] Optionally, in this embodiment, feature splicing is performed on the first input vector after feature extraction, the second input vector after feature extraction, and the third input vector after feature extraction. They can be converted into vectors in the same space through a domain mapping layer (FClayer) but are not limited to, and the three types of information are concatinated to obtain a media content vector of the second media information.

[0099] It should be noted that, when the audible discrimination model obtains the above-mentioned media content vector, it can, but is not limited to, classify the media content vector through multiple fully connected FC networks to obtain a determination result, wherein the determination result is used to indicate whether the second media information is audible.

[0100] As an optional solution, obtaining at least one media source information includes:

[0101] S1, obtaining at least one third media information, wherein the third media information is media information including text data, the non-audio data includes text data, and the media source information includes the third media information;

[0102] Obtaining audio applicable attributes associated with each piece of media source information in at least one piece of media source information, including:

[0103] S2: Obtain a second applicability attribute associated with each third media information in at least one third media information, wherein the second applicability attribute is used to indicate the applicability of the third media information when being played as audio.

[0104] Optionally, in this embodiment, the third media information may be, but is not limited to, media information containing text data, such as public account text, wherein at least one media source information may, but is not limited to, include the third media information.

[0105] Optionally, in this embodiment, when at least one media source information includes third media information, a second applicable attribute associated with the third media information is obtained, wherein the audio applicable attributes associated with each media source information in at least one media source information include the second applicable attribute associated with the third media information, and the second applicable attribute is used to indicate the adaptability of the third media information when played as audio.

[0106] As an optional solution, obtaining a second applicable attribute associated with each third media information in at least one third media information includes:

[0107] S1, obtaining author category information, text category information and text length information associated with each third media information, wherein the second applicable attribute includes the author category information, text category information and text length information;

[0108] Determining recommended media source information suitable for playing as audio from at least one piece of media source information by using the audio applicability attribute, and recommending the audio corresponding to the recommended media source information for playing, including:

[0109] S2, using author category information, text category information and text length information, determine from at least one third media information, second recommended media information suitable for playing as audio, and recommend the audio corresponding to the second recommended media information for playback, wherein the recommended media source information includes the second recommended media information.

[0110] Optionally, in this embodiment, the second applicable attribute can be determined based on but not limited to author category information, text category information and text length information associated with the third media information, wherein the author category information is used to indicate the author attribute information of the second media information, such as the author identity information of the public account article (commentator, narrator, writer, etc.); the text category information is used to indicate the text type information of the second media information, such as the text type of the public account article (film review, novel, essay, etc.); the text length information is used to indicate the text length information of the second media information, such as the text length value of the public account article (500 words, 800 words, etc.); and another example is the text length range of the public account article (0-50 words, 50-500 words, 500-5000 words, more than 5000 words, etc.).

[0111] Optionally, in this embodiment, the author category information, the text category information and the text length information are used to determine the second recommended media information suitable for playing as audio from at least one third media information.

[0112] It should be noted that in this embodiment, based on but not limited to the above-mentioned author category information, text category information and text length information, combined with the user's historical reading or historical listening habits, second media source information matching the user's reading habits or listening habits can be determined as the second recommended media information for recommended playback.

[0113] As an optional solution, the second recommended media information suitable for playing as audio is determined from at least one third media information by using the author category information, the text category information and the text length information, including:

[0114] S1, determining fourth media information of a target author category, fifth media information of a target text category, and sixth media information of a target text length from at least one third media information, wherein the target author category is an author category of historical third media information that the user has read within a historical time period, the target text category is a text category of the historical third media information, the target text length is a text length of the historical third media information, and the at least one third media information includes the fourth media information, the fifth media information, and the sixth media information;

[0115] The fourth media information, the fifth media information and the sixth media information are combined to obtain second recommended media information.

[0116] Optionally, in this embodiment, the fourth media information, the fifth media information, and the sixth media information may be, but are not limited to, a set that meets the screening condition and includes at least one piece of third media information.

[0117] Optionally, in this embodiment, the fourth media information, the fifth media information, and the sixth media information are processed as a union, which can be understood, but not limited to, as jointly determining the fourth media information, the fifth media information, and the sixth media information as the second recommended media information. For example, the fourth media information is set A, the fifth media information is set B, and the sixth media information is set C. Then, the second recommended media information obtained after the union processing is set D, and D = A∪B∪C.

[0118] Through the embodiments provided in the present application, based on the above-mentioned author category information, text category information and text length information, media information that hits user habits in multiple different dimensions is obtained, and then the union is calculated as the recommended content of the text type, thereby achieving the purpose of reasonably and scientifically improving the richness of the second recommended media information. It is not a blind recommendation, thereby achieving the technical effect of improving the richness of audio recommendations.

[0119] As an optional solution, the second recommended media information suitable for playing as audio is determined from at least one third media information by using the author category information, the text category information and the text length information, and may also include but is not limited to:

[0120] S1, determining fourth media information of a target author category, fifth media information of a target text category, and sixth media information of a target text length from at least one third media information, wherein the target author category is an author category of historical third media information that the user has read within a historical time period, the target text category is a text category of the historical third media information, the target text length is a text length of the historical third media information, and the at least one third media information includes the fourth media information, the fifth media information, and the sixth media information;

[0121] The fourth media information, the fifth media information and the sixth media information are subjected to intersection processing to obtain second recommended media information.

[0122] Optionally, in this embodiment, the fourth media information, the fifth media information, and the sixth media information may be, but are not limited to, a set that meets the screening condition and includes at least one piece of third media information.

[0123] Optionally, in this embodiment, the intersection processing is performed on the fourth media information, the fifth media information and the sixth media information, which can be understood, but not limited to, as determining the overlapping media information of the fourth media information, the fifth media information and the sixth media information as the second recommended media information. For example, the fourth media information is set A, the fifth media information is set B, and the sixth media information is set C, then the second recommended media information obtained after the union processing is set E, and E = A∩B∩C.

[0124] Through the embodiments provided in the present application, based on the above-mentioned author category information, text category information and text length information, media information that hits user habits in multiple different dimensions is obtained, and then the intersection is calculated as the recommended content of the text type, thereby achieving the technical effect of prioritizing the accuracy of the second recommended media information rather than the quantity of the second recommended media information, thereby achieving the accuracy of audio recommendations.

[0125] As an optional solution, before obtaining the audio applicable attribute associated with each media source information in the at least one media source information, the method further includes:

[0126] S1, performing security detection on data involved in playing each media source information as audio, wherein the media source information that passes the security detection is allowed to be recommended for playing as audio.

[0127] Optionally, in this embodiment, a security check is performed on the data involved in playing each media source information as audio, which can be but is not limited to removing the media source information of various security factors (such as media source information containing uncivilized language).

[0128] It should be noted that if the media source information itself is directly subjected to security detection and filtering, it will cause a waste of audio resources. For example, for media source information containing image data and audio data, if there are unsafe or non-compliant images in the image data but the audio data is safe and compliant normal audio, if the media source information is directly subjected to security detection, the media source information (including its image data and audio data) will be directly eliminated and filtered. However, by using the above method of this embodiment, a security check is performed on the data involved when the media source information is played as audio, that is, a security check is performed on the audio data included in the media source information and / or the audio data converted from the media source information, thereby avoiding the above-mentioned problem of audio resource waste.

[0129] As an optional solution, the method further includes:

[0130] S1, obtaining whitelist media information from a whitelist media database, wherein the applicability of the media information in the whitelist media database when played as audio is set to be higher than a preset threshold, and the preset threshold is a minimum threshold for recommended playback;

[0131] S2, recommending and playing the audio corresponding to the whitelist media information.

[0132] Optionally, in this embodiment, whitelist media information is obtained from a whitelist media database, and audio corresponding to the whitelist media information is recommended for playback, wherein the applicability of the media information in the whitelist media database when played as audio is set to be higher than a preset threshold, and the preset threshold is the minimum threshold for the playback.

[0133] Optionally, in this embodiment, whitelist creator information can also be obtained from a whitelist creator database, but is not limited to, and media information corresponding to the whitelist creator information (i.e., content published by the whitelist creator) can be stored in the whitelist media database, wherein the adaptability of the media information corresponding to the whitelist creator information is set by default to be higher than a preset threshold, and whitelist creators can include, but are not limited to, professional audio creators.

[0134] As an optional solution, the above-mentioned audio recommendation method is applied to the scenario of building a suitable audio pool based on the video pool and the text pool, so as to solve the problem that the traditional audio playback platform only records the audio created by professionals. Although the professionalism and suitability of listening can be guaranteed, the coverage and diversity of the materials are often far from meeting the needs of users, and it is difficult to meet the different interests of a large number of users.

[0135] Specifically, in this embodiment, first, an audible audio pool is constructed based on a video recommendation content pool, wherein the video recommendation content pool may be, but is not limited to, a pool for storing video recommendation content whose video content changes very quickly. Considering the subsequent processing performance of audio recommendation, a logic of updating the pool every k minutes (20-30 minutes) is adopted. The specific logic here is as follows:

[0136] In the Tth round of operation, the current video content recommendation pool and the qualified audio obtained in the previous cycle T-1 are first mixed together and deduplicated. All videos after deduplication are security filtered to remove videos that do not meet the requirements due to various security factors (such as uncivilized language), blacklisted by operations (such as blacklisting and filtering of video categories and author categories), etc.

[0137] The text results of the sound in the video are recognized by ASR technology (Automatic Speech Recognition, referred to as ASR). The number of ASR texts in the video M, the duration of ASR text M1, and the total duration of the video M2 are obtained. In order to select the video suitable for listening, the above results are used to formulate filtering rules, which can include but are not limited to:

[0138] Rule 1: M1 / M2>0.5, which means that the duration of the ASR segment in the video must exceed half of the entire video to ensure that there is enough listening content.

[0139] Rule 2: M / M1>=5 / s. The average ASR density within the ASR segment must be more than 5 words / second. This prevents some videos that are not suitable for listening due to too few words from entering the candidate pool.

[0140] In addition, some other video attributes will be filtered, such as the number of characters in the video title must be greater than 5 and the number of characters in the video description must be greater than 5, to prevent some meaningless videos from being mixed in. Such pre-rule filtering can reduce the data scale of model judgment, speed up calculation speed, and reduce the consumption of computing resources.

[0141] This batch of feeds is then input into the audible judgment neural network model for calculation.

[0142] It should be noted that the video information includes the audio information of the video and the image information of each frame of the video. In order to simplify the complexity of the model, it can be assumed that the internal frames of the video have little effect on whether it is audible, but the cover frame of the video will be the cover frame of the audio and will have an impact on the user's decision, so the cover frame information is selected as the input. The text information expressed by the audio information is very critical to whether the video is audible, so the ASR video text information obtained in the previous step is used as the input signal. Combined with the audio signal, a multimodal neural network containing visual information, auditory information, and text information is constructed to determine whether the video can be converted into audio suitable for listening.

[0143] Specifically, the specific network structure of the audible determination neural network model based on the above audio recommendation method is as follows: Figure 3 As shown, the audio input signal 302 is processed by logmel spectrum transformation (an audio feature extraction method that can better describe the frequency distribution characteristics of audio) and converted into a neural network input vector. The ASR text input signal 304 is embedded by word / character vector to obtain the input vector of the text. The video cover image information 306 is converted into an RGB three-channel image intensity expression vector. The audio signal is extracted by a bidirectional GRU recurrent neural network, and the text signal is extracted by a multi-layer Transformer structure. The image signal is extracted by a multi-layer CNN convolutional neural network. The three-way feature vectors are converted to vectors in the same space through a domain mapping layer (FClayer). The three types of information are concatenated (concat) to obtain the expression vector of the content. Classification is performed after passing through a multi-layer fully connected FC network.

[0144] Optionally, the initial training data of the audible model comes from the audible sample data determined by strong rules (whitelist authors, etc.) and the audible sample data with sampling and annotation. As the system runs, a method of intensive training is adopted: samples with a model-determined scoring threshold higher than x (for example, 0.9) are treated as positive samples, and samples with a scoring threshold lower than y (for example, 0.1) are treated as negative samples and mixed with annotated samples for mixed training. At the same time, some model scoring samples will be sampled and reviewed, and if there are any errors, they will be corrected before training.

[0145] Finally, the system blacklist is filtered to delete authors and videos (or data converted into audio) that are blacklisted due to various security and operational factors.

[0146] After obtaining the video recommendation content screening pool, it is cached and reused as the qualified audio at time T at time T+1. At the same time, it is written into the online kv storage table for calling.

[0147] Optionally, in this embodiment, in addition to building an audible audio pool based on the video recommendation content pool, an audible audio pool can also be built based on the platform operation whitelist creators, but is not limited to, that is, the platform needs to cultivate professional audio creators to supplement the creation. Here, it can be but is not limited to pulling the published content of the platform operation whitelist creators to add to the candidate pool, and mixing the video conversion content for calling.

[0148] Optionally, in this embodiment, in addition to constructing an audible audio pool based on the video recommendation content pool, audio can also be obtained based on, but not limited to, the conversion of text content in the official account. It is possible but not limited to select a certain range of high-quality authors through operational means, such as adopting some general rules, using author category restrictions (such as knowledge explanation, news information, technology talk, etc.), article category restrictions, and article length filtering restrictions. Suitable articles are collectively referred to as audio content through TTS (TextToSpeech) technology. As an important supplement to the audio content pool, they are also written into the online kv repository after blacklist filtering.

[0149] Optionally, in this embodiment, audio content is extracted from video content, and converted audio content is obtained from public account content, and they are converted into a unified audio ID system for subsequent recommendation process, and the content that best suits the user's personalized preferences is selected for recommendation, and the audio content is cached in the background storage for online call.

[0150] To further illustrate, a recommendation method based on the above audio recommendation method for constructing an audio pool suitable for listening based on a video pool and a text pool, such as Figure 4 As shown, including:

[0151] Step S402, merging the previously cached audible content pool: mixing the current video content recommendation pool and the qualified audio obtained in the previous cycle T-1 together.

[0152] Step S404, all videos are subjected to security filtering, video category filtering and author category filtering: various security factors (such as uncivilized language), operation blacklisting (such as video category and author category blacklisting filtering) and other videos that do not meet the requirements are removed.

[0153] Step S406, calculate the ASR recognition result of the video, and obtain ASR filtering information such as the length of the video ASR text, the duration of ASR, and the total duration of the video: the text result of the sound in the video is recognized by ASR technology (Automatic Speech Recognition, referred to as ASR). The number of video ASR texts M, the duration of ASR texts M1, and the total duration of the video M2 are obtained. In order to select a video suitable for listening, the above filtering rules are formulated using the above results, which will not be repeated here.

[0154] Step S408, audible model determination: the video that passes the above filtering rules is input into the audible neural network model for calculation and determination to determine whether the video can be converted into audio suitable for listening.

[0155] Step S410, blacklist filtering: perform system blacklist filtering to delete authors and videos that are blacklisted due to various security and operational factors (or the data involved in the conversion into audio).

[0156] Step S412, cache the current audible content pool snapshot: after obtaining the video recommendation content screening pool, cache it as the qualified audio at time T, and reuse it at time T+1.

[0157] Step S414, write kv storage for online calling.

[0158] Step S416, pull the video and convert it to obtain audio.

[0159] Step S418, fetch the published content of the whitelisted authors in the last 14 days.

[0160] Step S420, security filtering of the published content: deleting authors and published content that are blacklisted due to various security and operational factors.

[0161] Step S422, filtering the text category, author category, and word count of the authors within the scope.

[0162] Step S424, converting the audio using text-to-speech technology.

[0163] Step S426, blacklist filtering: perform system blacklist filtering to delete authors and audios that are blacklisted due to various security and operational factors.

[0164] Through the embodiments provided by this application, a recommendation method for building an audio pool suitable for listening based on a huge video pool and a text pool is designed, and it can be applied to relevant audio playback platforms for audio recommendation. As the basis of the entire upper-level recommendation application, a background listening channel is added on the basis of the current video swiping method, solving the user's demand for listening. This embodiment constructs audio data from a video pool and a text pool. On the one hand, it solves the problem of the source of materials for audio content, proposes a complete process for building an audible audio pool, and proposes a deep network model for audio audibility determination that integrates multimodal features of audio, text, and images, so that the determination of audio audibility can be automated and highly accurate, and exceeds the capabilities that a simple audio recommendation platform can provide in terms of quantity, coverage of user interests, and scalability. On the other hand, audio recommendations can be made to users through the recommended interest content of users in videos and articles, which can effectively solve the problem of inaccurate capture of user interests in the cold start stage of the scene, and give users the possibility of content recommendations that surprise them.

[0165] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0166] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0167] According to another aspect of the embodiments of the present application, an audio recommendation device for implementing the above audio recommendation method is also provided. Figure 5 As shown, the device comprises:

[0168] A first acquisition unit 502 is configured to acquire at least one piece of media source information, wherein the media source information is media information including at least one piece of non-audio data;

[0169] A second acquisition unit 504 is used to acquire an audio applicability attribute associated with each media source information in at least one media source information, wherein the audio applicability attribute is used to indicate the applicability of the media source information when it is played as audio;

[0170] The determination unit 506 is used to determine, from at least one piece of media source information, recommended media source information suitable for playing as audio by using the audio applicability attribute, and recommend playing the audio corresponding to the recommended media source information.

[0171] For specific embodiments, reference may be made to the example shown in the audio recommendation device, which will not be described in detail in this example.

[0172] As an optional solution, the first acquisition unit 502 includes: a first acquisition module, configured to acquire at least one first media information, wherein the first media information is media information including image data and audio data, the non-audio data includes image data, and the media source information includes the first media information;

[0173] The second acquisition unit 504 includes: a second acquisition module, which is used to acquire a first applicability attribute associated with each first media information in at least one first media information, wherein the first applicability attribute is used to indicate the applicability of the first media information when it is played as audio.

[0174] For the specific implementation, reference may be made to the example shown in the audio recommendation method, which will not be described in detail in this example.

[0175] As an optional solution, the second acquisition module includes: a recognition submodule, which is used to perform voice recognition processing on at least one first media information respectively to obtain first text data corresponding to each first media information, wherein the first text data is text data after the first media information is converted into text, and the first applicable attribute includes the first text data;

[0176] The determination unit 506 includes: a first determination module, which is used to use the first text data to determine the first recommended media information suitable for playing as audio from at least one first media information, and recommend the audio corresponding to the first recommended media information for playing, wherein the recommended media source information includes the first recommended media information.

[0177] For the specific implementation, reference may be made to the example shown in the audio recommendation method, which will not be described in detail in this example.

[0178] As an optional solution, the first determining module is used to include:

[0179] The first determination submodule is used to determine, when the first text data includes first text density data and second text density data, first recommended media information from at least one first media information using the first text density data and the second text density data, wherein the first text density data is greater than a first preset threshold and the second text density data is greater than a second preset threshold, wherein the first text density data is used to represent the density of the first duration distributed over the second duration, the second text density data is used to represent the density of the text data after the first media information is converted into text over the first duration, the first duration is used to represent the duration occupied by the first text data in each corresponding first media information, and the second duration is used to represent the duration of each first media information corresponding to the first text data.

[0180] For the specific implementation, reference may be made to the example shown in the audio recommendation method, which will not be described in detail in this example.

[0181] As an optional solution, the first determination module includes:

[0182] A first screening submodule, configured to screen at least one first media information using the first text data to obtain at least one second media information, wherein the at least one first media information includes at least one second media information;

[0183] a determination submodule, configured to input each second media information of the at least one second media information into the audible determination model for calculation, and obtain a determination result of each second media information;

[0184] The second screening submodule is used to use the judgment result to perform a secondary screening on at least one second media information to obtain recommended media source information, wherein the recommended media source information is the second media information indicated as audible by the judgment result, and the audible discrimination model is a neural network model pre-trained based on audible positive sample data and inaudible negative sample data, and is used to determine whether the input second media information is audible.

[0185] For the specific implementation, reference may be made to the example shown in the audio recommendation method, which will not be described in detail in this example.

[0186] As an optional solution, the determination submodule includes:

[0187] An input subunit, used to input the audio input signal, text input signal and video cover image information of each second media information into the audible discrimination model;

[0188] a transform subunit, configured to perform spectrum transform processing on the audio input signal to obtain a first input vector, perform text transform processing on the text input signal to obtain a second input vector, and perform image transform processing on the video cover image information to obtain a third input vector;

[0189] The splicing subunit is used to perform feature splicing processing on the first input vector after feature extraction, the second input vector after feature extraction, and the third input vector after feature extraction to obtain media content vectors of each second media information, wherein the audible discrimination model performs classification and judgment processing based on the media content vectors to obtain judgment results of each second media information.

[0190] For the specific implementation, reference may be made to the example shown in the audio recommendation method, which will not be described in detail in this example.

[0191] As an optional solution, the first acquisition unit 502 includes: a third acquisition module, configured to acquire at least one third media information, wherein the third media information is media information including text data, the non-audio data includes text data, and the media source information includes the third media information;

[0192] The second acquisition unit 504 includes: a fourth acquisition module, which is used to acquire a second applicability attribute associated with each third media information in at least one third media information, wherein the second applicability attribute is used to indicate the applicability of the third media information when it is played as audio.

[0193] For specific embodiments, reference may be made to the example shown in the audio recommendation device, which will not be described in detail in this example.

[0194] As an optional solution, the fourth acquisition module includes: an acquisition submodule, which is used to acquire author category information, text category information and text length information associated with each third media information, wherein the second applicable attribute includes the author category information, the text category information and the text length information;

[0195] The determination unit 506 includes: a second determination module, which is used to use the author category information, the text category information and the text length information to determine, from at least one third media information, second recommended media information suitable for playing as audio, and recommend the audio corresponding to the second recommended media information for playback, wherein the recommended media source information includes the second recommended media information.

[0196] For specific embodiments, reference may be made to the example shown in the audio recommendation device, which will not be described in detail in this example.

[0197] As an optional solution, the second determining module includes:

[0198] A second determination submodule is used to determine fourth media information of a target author category, fifth media information of a target text category, and sixth media information of a target text length from at least one third media information, wherein the target author category is an author category of historical third media information that the user has read within a historical time period, the target text category is a text category of the historical third media information, the target text length is a text length of the historical third media information, and the at least one third media information includes the fourth media information, the fifth media information, and the sixth media information;

[0199] The third determination submodule is used to perform a union process on the fourth media information, the fifth media information and the sixth media information to obtain second recommended media information.

[0200] For specific embodiments, reference may be made to the example shown in the audio recommendation device, which will not be described in detail in this example.

[0201] As an optional solution, the device also includes:

[0202] The detection module is used to perform security detection on data involved in playing each media source information as audio before obtaining the audio applicability attributes associated with each media source information in at least one media source information, wherein the media source information that passes the security detection is allowed to be recommended for playback as audio.

[0203] As an optional solution, the device also includes:

[0204] The fifth acquisition module is used to obtain whitelist media information from a whitelist media database, wherein the applicability of the media information in the whitelist media database when played as audio is set to be higher than a preset threshold, and the preset threshold is the minimum threshold for recommended playback; the recommended playback module is used to recommend the audio corresponding to the whitelist media information for playback.

[0205] For specific embodiments, reference may be made to the example shown in the audio recommendation device, which will not be described in detail in this example.

[0206] According to another aspect of the embodiment of the present application, an electronic device for implementing the above audio recommendation method is also provided. The electronic device can be but is not limited to Figure 1 The client 102 or server 112 shown in FIG. 1 is a client 102 or a server 112 shown in FIG. 1 . In this embodiment, the electronic device is used as the client 102 as an example. Figure 6 As shown, the electronic device includes a memory 602 and a processor 604. The memory 602 stores a computer program, and the processor 604 is configured to execute the steps in any of the above method embodiments through the computer program.

[0207] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0208] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0209] S1, obtaining at least one piece of media source information, wherein the media source information is media information including at least one piece of non-audio data;

[0210] S2, obtaining an audio applicability attribute associated with each piece of media source information in at least one piece of media source information, wherein the audio applicability attribute is used to indicate the applicability of the media source information when it is played as audio;

[0211] S3, using the audio applicability attribute, determining recommended media source information suitable for playing as audio from at least one media source information, and recommending the audio corresponding to the recommended media source information for playing.

[0212] Alternatively, a person skilled in the art may understand that: Figure 6 The structure shown is for illustration only. Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 6 Different configurations are shown.

[0213] Among them, the memory 602 can be used to store software programs and modules, such as the program instructions / modules corresponding to the audio recommendation method and device in the embodiments of the present application. The processor 604 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, that is, realizing the above-mentioned audio recommendation method. The memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 602 may further include a memory remotely located relative to the processor 604, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 602 can be specifically, but not limited to, used to store information such as audio applicability attributes. As an example, such as Figure 6 As shown, the memory 602 may include but is not limited to the first acquisition unit 502, the second acquisition unit 504, and the determination unit 506 in the audio recommendation device. In addition, it may also include but is not limited to other module units in the audio recommendation device, which will not be repeated in this example.

[0214] Optionally, the transmission device 606 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 606 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 606 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0215] In addition, the electronic device further includes: a display 608 for displaying information such as audio applicable attributes; and a connection bus 610 for connecting various module components in the electronic device.

[0216] In other embodiments, the client or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. A peer-to-peer network may be formed between the nodes, and any form of computing device, such as a server, a client, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0217] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0218] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0219] It should be noted that the computer system of the electronic device is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0220] The computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM). In the random access memory, various programs and data required for system operation are also stored. The central processing unit, the read-only memory and the random access memory are connected to each other through a bus. The input / output interface (I / O interface) is also connected to the bus.

[0221] The following components are connected to the input / output interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a local area network card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the input / output interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0222] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions defined in the system of the present application are executed.

[0223] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0224] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0225] S1, obtaining at least one piece of media source information, wherein the media source information is media information including at least one piece of non-audio data;

[0226] S2, obtaining an audio applicability attribute associated with each piece of media source information in at least one piece of media source information, wherein the audio applicability attribute is used to indicate the applicability of the media source information when it is played as audio;

[0227] S3, using the audio applicability attribute, determining recommended media source information suitable for playing as audio from at least one media source information, and recommending the audio corresponding to the recommended media source information for playing.

[0228] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the electronic device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0229] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0230] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0231] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0232] In the several embodiments provided in the present application, it should be understood that the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0233] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0235] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An audio recommendation method, characterized in that: include: Acquire at least one piece of media source information, wherein the media source information is media information including at least one piece of non-audio data; Acquire an audio applicability attribute associated with each piece of media source information in the at least one piece of media source information, wherein the audio applicability attribute is used to indicate the applicability of the media source information when it is played as audio; By utilizing the audio applicability attribute, recommended media source information suitable for playing as audio is determined from the at least one media source information, and the audio corresponding to the recommended media source information is recommended for playing.

2. The method according to claim 1, characterized in that The acquiring of at least one media source information includes: acquiring at least one first media information, wherein the first media information is media information including image data and audio data, the non-audio data includes the image data, and the media source information includes the first media information; The obtaining of the audio applicability attributes associated with each piece of media source information in the at least one media source information includes: obtaining a first applicability attribute associated with each piece of first media information in the at least one first media information, wherein the first applicability attribute is used to indicate the applicability of the first media information when being played as audio.

3. The method according to claim 2, characterized in that The obtaining of the first applicable attribute associated with each first media information in the at least one first media information includes: performing speech recognition processing on each of the at least one first media information to obtain first text data corresponding to each of the first media information, wherein the first text data is text data after the first media information is converted into text, and the first applicable attribute includes the first text data; The method of utilizing the audio applicability attribute to determine, from the at least one media source information, recommended media source information suitable for playing as audio, and recommending the audio corresponding to the recommended media source information for playing, includes: utilizing the first text data to determine, from the at least one first media information, first recommended media information suitable for playing as audio, and recommending the audio corresponding to the first recommended media information for playing, wherein the recommended media source information includes the first recommended media information.

4. The method according to claim 3, characterized in that The using the first text data to determine the first recommended media information suitable for playing as audio from the at least one first media information includes: In the case that the first text data includes first text density data and second text density data, the first text density data and the second text density data are used to determine, from the at least one first media information, the first recommended media information in which the first text density data is greater than a first preset threshold and the second text density data is greater than a second preset threshold, wherein the first text density data is used to represent the density of the first duration distributed over the second duration, and the second text density data is used to represent the density of the text data after the first media information is converted into text over the first duration, the first duration is used to represent the duration occupied by the first text data in the corresponding respective first media information, and the second duration is used to represent the duration of the respective first media information corresponding to the first text data.

5. The method according to claim 3, characterized in that: The using the first text data to determine the first recommended media information suitable for playing as audio from the at least one first media information includes: Using the first text data, screening the at least one first media information to obtain at least one second media information, wherein the at least one first media information includes the at least one second media information; Inputting each second media information of the at least one second media information into the audible determination model for calculation, and obtaining a determination result of each second media information; Using the determination result, the at least one second media information is secondary screened to obtain the recommended media source information, wherein the recommended media source information is the second media information that the determination result indicates is audible, and the audible discrimination model is a neural network model pre-trained based on audible positive sample data and inaudible negative sample data, and is used to determine whether the input second media information is audible.

6. The method according to claim 5, characterized in that The step of inputting each of the second media information into an audible determination model for calculation to obtain a determination result of each of the second media information includes: Inputting the audio input signal, text input signal and video cover image information of each second media information into the audible discrimination model; Performing spectrum transformation processing on the audio input signal to obtain a first input vector, performing text transformation processing on the text input signal to obtain a second input vector, and performing image transformation processing on the video cover image information to obtain a third input vector; The first input vector after feature extraction, the second input vector after feature extraction, and the third input vector after feature extraction are subjected to feature splicing processing to obtain media content vectors of the respective second media information, wherein the audible discrimination model performs classification and determination processing based on the media content vectors to obtain the determination results of the respective second media information.

7. The method according to claim 1, characterized in that The acquiring of at least one media source information comprises: acquiring at least one third media information, wherein the third media information is media information including text data, the non-audio data includes the text data, and the media source information includes the third media information; The obtaining of the audio applicability attribute associated with each piece of media source information in the at least one media source information includes: obtaining a second applicability attribute associated with each piece of third media information in the at least one third media information, wherein the second applicability attribute is used to indicate the applicability of the third media information when played as audio.

8. The method according to claim 7, characterized in that The obtaining of the second applicable attribute associated with each third media information in the at least one third media information comprises: obtaining author category information, text category information and text length information associated with each third media information, wherein the second applicable attribute comprises the author category information, the text category information and the text length information; The method of utilizing the audio applicability attribute to determine, from the at least one media source information, recommended media source information suitable for playing as audio, and recommending the audio corresponding to the recommended media source information for playing, includes: utilizing the author category information, the text category information, and the text length information to determine, from the at least one third media information, second recommended media information suitable for playing as audio, and recommending the audio corresponding to the second recommended media information for playing, wherein the recommended media source information includes the second recommended media information.

9. The method according to claim 8, characterized in that The step of determining, from the at least one third media information, second recommended media information suitable for playing as audio by using the author category information, the text category information, and the text length information includes: Determining fourth media information of a target author category, fifth media information of a target text category, and sixth media information of a target text length from the at least one third media information, wherein the target author category is an author category of historical third media information that the user has read within a historical time period, the target text category is a text category of the historical third media information, the target text length is a text length of the historical third media information, and the at least one third media information includes the fourth media information, the fifth media information, and the sixth media information; Performing a union process on the fourth media information, the fifth media information and the sixth media information to obtain the second recommended media information.

10. The method according to any one of claims 1 to 9, characterized in that: Before acquiring the audio applicable attribute associated with each piece of media source information in the at least one piece of media source information, the method further includes: A security check is performed on data involved in playing each piece of media source information as audio, wherein the media source information that passes the security check is allowed to be recommended and played as audio.

11. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Acquire whitelist media information from a whitelist media database, wherein the applicability of the media information in the whitelist media database when played as audio is set to be higher than a preset threshold, and the preset threshold is a minimum threshold for the recommended playback; The audio corresponding to the whitelist media information is recommended for playing.

12. An audio recommendation device, characterized in that: include: A first acquisition unit, configured to acquire at least one piece of media source information, wherein the media source information is media information including at least one piece of non-audio data; a second acquisition unit, configured to acquire an audio applicability attribute associated with each piece of media source information in the at least one piece of media source information, wherein the audio applicability attribute is used to indicate the applicability of the media source information when it is played as audio; The determination unit determines, by using the audio applicability attribute, recommended media source information suitable for playing as audio from the at least one media source information, and recommends playing the audio corresponding to the recommended media source information.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 11 when executed by an electronic device.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

15. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 11 through the computer program.

Images