Audio and video transcoding processing and playback method, device, storage medium and program product

Abstract

The embodiment of the present application provides an audio and video transcoding processing and playback method, device, storage medium and program product, which relates to the field of cloud computing technology. In the embodiment of the present application, for the target frame that fails in the transcoding process of the audio and video content, a corresponding supplementary frame is generated according to the transcoding parameters corresponding to the target frame; the supplementary frame is filled into the transcoding position corresponding to the target frame to serve as the transcoding result frame corresponding to the target frame, thereby solving the problem of missing transcoding result frames in the transcoded audio and video content. In this way, even if part of the audio and video content cannot be transcoded, the transcoded audio and video content with a complete number of frames can be obtained, ensuring the continuity of the playback of the transcoded audio and video content, and there will be no problems such as termination of playback or resetting of playback progress, and the user does not need to manually drag the progress bar to skip the transcoding failed segment, which is conducive to improving the user's playback experience.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to an audio and video transcoding processing and playback method, device, storage medium and program product. Background Art

[0002] In the fields of video on demand or live broadcast, the compressed and encoded original audio and video content is usually transcoded into the target audio and video content to adapt to different network bandwidths, different terminal processing capabilities and different user needs. Common transcoding methods include full offline transcoding and transcoding while broadcasting.

[0003] In the full offline transcoding method, the original audio and video content is fully transcoded in advance. If part of the audio and video content cannot be transcoded, the entire transcoding process will be terminated, and the subsequent audio and video content will not be transcoded. When watching and playing, the end user can only play the audio and video content that has been successfully transcoded, and the subsequent audio and video content that has not been transcoded will not be played, resulting in a poor user experience.

[0004] In the transcoding while playing mode, the audio and video clips requested for playback are transcoded on demand according to the playback request of the terminal user. If the audio and video clips cannot be transcoded, the player will terminate the playback, or the player will crash, or the playback progress will be reset, or the user will need to manually drag the progress bar to skip the clip before continuing to play, resulting in a poor user experience.

[0005] Therefore, whether it is the full offline transcoding method or the transcoding while broadcasting method, when some audio and video content cannot be transcoded, it will cause poor playback experience for end users. Summary of the invention

[0006] Multiple aspects of the present application provide an audio and video transcoding processing and playback method, device, storage medium and program product to solve the problem of poor user playback experience caused by the inability to transcode some audio content, thereby improving the user playback experience.

[0007] The embodiment of the present application provides an audio and video transcoding processing method, including: obtaining audio and video content to be transcoded and its corresponding target transcoding parameters; transcoding the audio and video content to be transcoded according to the target transcoding parameters; determining a target frame in which transcoding fails in the audio and video content to be transcoded, and generating a supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame in the target transcoding parameters; adding the supplementary frame as a transcoding result frame of the target frame to the transcoding position corresponding to the target frame, so as to obtain the transcoded audio and video content corresponding to the audio and video content to be transcoded.

[0008] The embodiment of the present application also provides an audio and video playback method, including: sending a playback request to a media manager to request the media manager to transcode the audio and video content, wherein the transcoded audio and video content is obtained by transcoding the audio and video content to be transcoded; receiving the transcoded audio and video content sent by the media manager, wherein the transcoded audio and video content includes a transcoding result frame, and the transcoding result frame at least includes a supplementary frame corresponding to a target frame determined to have failed to be transcoded in the audio and video content to be transcoded; and playing the transcoding result frame in the transcoded audio and video content.

[0009] An embodiment of the present application also provides a media manager, comprising: a memory and a processor; the memory is used to store a computer program, and the processor is coupled to the memory and is used to execute the computer program to implement the steps in the above method.

[0010] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above method.

[0011] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0012] In the embodiment of the present application, for the target frame that fails in the transcoding process of the audio and video content, a corresponding supplementary frame is generated according to the transcoding parameters corresponding to the target frame; the supplementary frame is filled into the transcoding position corresponding to the target frame as the transcoding result frame corresponding to the target frame, solving the problem of missing transcoding result frames in the transcoded audio and video content. In this way, even if part of the audio and video content cannot be transcoded, the transcoded audio and video content with a complete number of frames can be obtained, ensuring the continuity of the playback of the transcoded audio and video content, and there will be no problems such as termination of playback or resetting of playback progress, and the user does not need to manually drag the progress bar to skip the transcoding failed segment, which is conducive to improving the user's playback experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1 A schematic diagram of a flow chart of audio and video transcoding processing provided for an exemplary embodiment of the present application;

[0015] Figure 2 A flowchart of an audio processing method provided by an exemplary embodiment of the present application;

[0016] Figure 3 A schematic diagram of the structure of an audio sequence and a video sequence provided for another exemplary embodiment of the present application;

[0017] Figure 4 A structural diagram of an audio and video transcoding processing system provided for an exemplary embodiment of the present application;

[0018] Figure 5 A schematic diagram of a process of supplementing frame filling provided by another exemplary embodiment of the present application;

[0019] Figure 6 A structural diagram of an audio and video transcoding processing system provided for another exemplary embodiment of the present application;

[0020] Figure 7 A structural diagram of a media manager provided for another exemplary embodiment of the present application;

[0021] Figure 8 A schematic structural diagram of an electronic device is provided as yet another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0023] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0024] Audio and video transcoding refers to the process of converting original audio content from one encoding format to another. During the transcoding process, transcoding failure may occur, which results in discontinuity of the transcoded audio and video content, i.e., missing some of the original audio frames or transcoded result frames corresponding to the original video frames, which in turn results in the inability to play the transcoded audio and video content continuously, reducing the user's playback experience.

[0025] To this end, in an embodiment of the present application, a new audio and video transcoding process is provided. During the transcoding process of the audio and video content to be transcoded, if a target frame that fails to be transcoded is found, a supplementary frame corresponding to the target frame is generated according to the transcoding parameters corresponding to the target frame, and the supplementary frame is added as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame, so as to obtain the transcoded audio and video content with a complete number of frames, thereby ensuring the continuity of the playback of the transcoded audio and video content, and preventing problems such as termination of playback or resetting of playback progress. The user does not need to manually drag the progress bar to skip the transcoding failure segment, which is conducive to improving the user's playback experience. The audio and video transcoding process of the present application is described in detail below in conjunction with the detailed embodiments.

[0026] Figure 1 The following is a flowchart of an audio and video transcoding process provided by an exemplary embodiment of the present application. Figure 1 As shown in the figure, the audio and video transcoding process mainly includes the following steps:

[0027] S1: Decapsulation: separating the uploaded audio and video content to be transcoded into an audio stream and a video stream, wherein the audio and video content to be transcoded has a first encoding format;

[0028] S2: Decoding: Decoding is to convert the compressed audio stream into uncompressed audio raw data, and to convert the compressed video stream data into uncompressed video raw data;

[0029] S3: Pre-processing before encoding: Before re-encoding, various processing is performed on the decoded original audio data and original video data, such as resolution conversion and frame rate conversion for the original video data, and audio sampling rate and channel conversion for the original audio data;

[0030] S4: encoding: encoding and compressing the processed audio data and the processed video data into a second encoding format;

[0031] S5: Encapsulation: Packaging the encoded and compressed audio data and video data into transcoded audio and video content to facilitate transmission and playback.

[0032] In this embodiment, the audio and video content to be transcoded may be audio and video content having a first encoding format uploaded by a content provider, and the audio and video content to be transcoded may be, but is not limited to, an audio and video file or a partial audio and video clip in an audio and video file. From the perspective of the type of audio and video content, the audio and video content to be transcoded may be, but is not limited to: short videos, TV series, variety shows, audio and video advertisements, etc. Among them, the audio and video content to be transcoded includes audio frames and video frames, and the number of audio frames and video frames may be one frame or more frames, depending on the length of the audio and video content to be transcoded. In this embodiment, the audio frames and video frames are decoded, preprocessed, and encoded, respectively, and finally the encoding results of the audio frames and video frames are encapsulated to obtain the transcoded audio and video content. The present embodiment does not limit the encoding format of the audio and video content to be transcoded, for example, it may be but not limited to H.264 (Advanced Video Coding), H.265 (High Efficiency Video Coding), AV1 (AOMedia Video 1, an open film encoding format), etc. For the convenience of description and distinction, the encoding format of the audio and video content to be transcoded is referred to as the first encoding format, and the encoding format of the transcoded audio and video content is referred to as the second encoding format. The second encoding format may be another encoding format different from the first encoding format. For example, the first encoding format is H.264, and the second encoding format may be H.265 or AV1, etc.; for another example, the first encoding format is H.256, and the second encoding format is H.264 or AV1, etc.

[0033] like Figure 1 As shown, the audio and video transcoding process provided in this embodiment includes but is not limited to Figure 1 Steps S1-S5 shown. During the audio and video transcoding process, failure may occur in a certain step, for example, failure may occur in the above-mentioned decoding step S2, failure may also occur in the decapsulation step S1, and of course failure may also occur in the video processing step (ie S3) and the encoding step S4, etc. The reasons for failure in different steps are generally different. In the embodiment of the present application, the reasons for the transcoding failure are not concerned, that is, the transcoding failure that occurs in any step is applicable to the embodiment of the present application. Taking into account that the audio frames and video frames are processed separately in the present embodiment, the audio and video content to be transcoded may fail to be transcoded in one or some audio frames, or in one or some video frames. For the sake of ease of description, the audio frames or video frames that fail to be transcoded are collectively referred to as target frames, that is, the target frames may be audio frames or video frames.

[0034] In this embodiment, a supplementary frame is generated for the target frame where transcoding fails, and the supplementary frame is added to the transcoding position corresponding to the target frame as the transcoding result frame of the target frame, so as to obtain the transcoded audio and video content with a complete number of frames. Among them, the complete number of frames means that the number of frames contained in the transcoded audio and video content is the same as the number of frames contained in the audio and video content to be transcoded, so that the transcoded audio and video content will not be discontinuous due to the failure of transcoding of the target frame, so that the transcoded audio and video content can be played continuously, and there will be no problems such as termination of playback or reset of playback progress or the need for the user to manually drag the progress bar to skip due to the missing transcoding result frame corresponding to the target frame. Among them, the playback progress reset means that each time the playback reaches the missing frame position, it will start playing from the beginning again, that is, it will play in a loop between the starting position and the missing frame position.

[0035] It is to be noted that in the embodiment of the present application, the role of the supplementary frame is to solve the problem of missing frames in the transcoded audio and video content due to transcoding failure, and to ensure that the frame content exists at the transcoding position corresponding to the target frame, rather than being missing or empty. There is no limitation on the content of the supplementary frame, and frames of any content can be used as supplementary frames in the embodiment of the present application. The transcoding position corresponding to the target frame in the embodiment of the present application refers to the position corresponding to the target frame in the transcoded audio and video content, and there is a one-to-one correspondence between the position of each frame in the audio and video content to be transcoded and the position of each frame in the transcoded audio and video content.

[0036] In addition, in the embodiments of the present application, the implementation timing of generating a supplementary frame for a target frame and adding the supplementary frame to the transcoding position corresponding to the target frame is not limited. The implementation timing here can be understood as the implementation position in the audio and video transcoding process, for example, it can be implemented during the execution of the step where the target frame is located or in a subsequent step. For example, assuming that a decoding failure occurs in the above-mentioned decoding step S2, wherein the audio frame or video frame that fails to be decoded is an example of a target frame, optionally, when a target frame that fails to be decoded occurs in step S2, a supplementary frame corresponding to the target frame can be directly generated, and the supplementary frame can be sent to the subsequent step S3 as the transcoding result frame at the transcoding position corresponding to the target frame for preprocessing, and then encoded in step S4, and finally encapsulated in step S5 for each frame encoded in step S4 to obtain the transcoded audio and video content. Alternatively, still taking the case where decoding fails in the above decoding step S2 as an example, the target frame where decoding fails can be determined in the preprocessing step S3, a supplementary frame is generated for the target frame, and the supplementary frame is sent to the subsequent step S3 as the transcoding result frame at the corresponding transcoding position of the target frame for preprocessing, and then encoded in step S4, and finally encapsulated in step S5 for each frame encoded in step S4 to obtain the transcoded audio and video content. Of course, the target frame where decoding fails can also be determined in the encoding step S4 or the encapsulation step S5, a supplementary frame is generated for the target frame, and the supplementary frame is used as the transcoding result frame at the corresponding transcoding position of the target frame. It should be noted that before the supplementary frame is generated for the target frame, the information of the target frame decoding failure can be sequentially transmitted to the subsequent steps until the supplementary frame is generated for the target frame. Among them, the decoding process is executed in sequence, and the decoding failure information will be output at the frame position where decoding fails or no information will be output (that is, the output is empty), and the information of "the decoding failure information will be output at the frame position where decoding fails or no information will be output" can be used as the information of the target frame decoding failure and can be sequentially transmitted to the subsequent steps.

[0037] Furthermore, in the embodiments of the present application, the execution subject of the audio and video transcoding processing method is not limited, and it can be in the terminal where the player is located, or in the server corresponding to the player. Optionally, the server can be deployed in the cloud. Further, when the server is deployed in the cloud, it can include a cloud storage node and a media manager. The media manager is responsible for completing the audio and video transcoding process. In the following embodiments, the media manager is used as an example for explanation.

[0038] The following is a detailed description of an audio and video transcoding method according to an embodiment of the present application. Figure 2 As shown, the method includes:

[0039] S201: Acquire the audio and video content to be transcoded and its corresponding target transcoding parameters;

[0040] S202: performing transcoding processing on the audio and video content to be transcoded according to the target transcoding parameters;

[0041] S203: determining a target frame that fails to be transcoded in the audio and video content to be transcoded, and generating a supplementary frame corresponding to the target frame according to a transcoding parameter corresponding to the target frame in the target transcoding parameter;

[0042] S204: adding the supplementary frame as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame, so as to obtain the transcoded audio and video content corresponding to the audio and video content to be transcoded.

[0043] In this embodiment, the audio and video content to be transcoded is part of the audio and video content or all of the audio and video content in the original audio and video content uploaded by the content provider. The audio and video content to be transcoded includes audio frames and video frames, the number of audio frames and video frames is one frame or more frames, and the audio and video content to be transcoded corresponds to target transcoding parameters, the target transcoding parameters include the transcoding parameters corresponding to the audio frames and the video frames respectively, and the transcoding parameters refer to the parameters that the transcoded audio and video frames need to meet, corresponding to the second transcoding format. For video frames, transcoding parameters may include but are not limited to: resolution, frame rate, codec (codec), etc.; for audio frames, transcoding parameters may include but are not limited to: sampling rate, number of channels, codec, etc.

[0044] like Figure 3 As shown, the audio and video content to be transcoded includes an audio sequence corresponding to an audio frame, and a video sequence corresponding to a video frame; the audio sequence may include one or more audio frames, Figure 3 In the example, the audio series includes 5 audio frames, from the 1st frame to the 5th frame; the video frame may include one or more video frames. Figure 3 In the example, a video sequence includes 5 video frames, from the 1st frame to the 5th frame, but it is not limited thereto. There is a corresponding relationship between the audio frame and the video frame, and the audio frame and the video frame are synchronized in time during playback, such as Figure 3 The first frame of the audio sequence corresponds to the first frame of the video sequence, and so on.

[0045] In this embodiment, the target frame refers to an audio frame or video frame in which the transcoding process of the audio and video content to be transcoded fails. The transcoding position corresponding to the target frame can be the frame position corresponding to the target frame in the transcoded audio and video content, and the frame position has a corresponding relationship with the frame position corresponding to the target frame in the audio and video content to be transcoded.

[0046] It should be noted that, during the entire transcoding process, the transcoding failure may include both audio frames and video frames, or only audio frames, or only video frames, but for a certain frame where the failure occurs, it can only be a video frame or an audio frame. The "frame" involved in the embodiments of the present application refers to an audio frame or a video frame.

[0047] Further, according to the transcoding parameters corresponding to the target frame in the target transcoding parameters, a supplementary frame corresponding to the target frame is generated; the supplementary frame is added as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame, so as to obtain the transcoded audio and video content corresponding to the audio and video content to be transcoded. In this embodiment, the transcoded audio and video content includes the transcoding result frames corresponding to each audio frame and video frame in the audio and video content to be transcoded, and these transcoding result frames include the transcoding result frames corresponding to the audio frames and / or video frames that are successfully transcoded, and also include the transcoding result frames (i.e., supplementary frames) corresponding to the audio frames and / or video frames that fail to be transcoded.

[0048] Among them, this embodiment does not limit the generation and addition timing of the supplementary frame. For example, during the transcoding process, whenever a transcoding failure is identified, a supplementary frame corresponding to the target frame where the transcoding failed is generated in real time, and the supplementary frame is added to the transcoding position corresponding to the target frame in the corresponding transcoding processing step; or, during the transcoding process, the frame number of the target frame where the transcoding failed is recorded, and after the entire transcoding process for the audio and video content to be transcoded is completed, a supplementary frame is generated for the target frame according to the recorded frame number of the target frame, and the supplementary frame is added to the audio and video content after the initial transcoding to obtain the final transcoded audio and video content. Among them, relative to the final transcoded audio and video content, the initial transcoded audio and video content is the audio and video content with missing frames obtained after the transcoding is completed.

[0049] In the embodiment of the present application, for the target frame that fails in the transcoding process of the audio and video content, a corresponding supplementary frame is generated according to the transcoding parameters corresponding to the target frame; the supplementary frame is filled into the transcoding position corresponding to the target frame as the transcoding result frame corresponding to the target frame, solving the problem of missing frames in the audio and video content after transcoding. In this way, even if part of the audio and video content cannot be transcoded, the transcoded audio and video content with a complete number of frames can be obtained, ensuring the continuity of the playback of the transcoded audio and video content, and there will be no problems such as termination of playback or resetting of playback progress, and the user does not need to manually drag the progress bar to skip the failed transcoding segment, which is conducive to improving the user's playback experience.

[0050] It is to be noted that the audio and video transcoding processing method provided by this solution can be applied to the scenario of simultaneous transcoding and broadcasting, and can also be applied to the scenario of full offline transcoding. The following introduces the technical solution of this application in combination with the scenario of simultaneous transcoding and broadcasting and the method of full offline transcoding.

[0051] Transcode and play simultaneously supports real-time transcoding of audio and video clips on demand during playback. Figure 4 The structure diagram of an audio and video transcoding processing system provided by an exemplary embodiment of the present application is as follows. Figure 4As shown, the system 400 includes: a cloud storage node 401, a content provider 402, a media player 403 at the user end, and a media manager 4011. The cloud storage node 401 is responsible for providing storage services for audio and video content to be transcoded and after transcoding, and both the audio and video content to be transcoded and after transcoding include at least one audio and video clip; the media player 403 runs on the terminal device at the user end, and is responsible for requesting the transcoded audio and video clip from the cloud storage node 401 for playback, and the media manager 4011 is responsible for transcoding the audio and video clip to be transcoded, providing a basis for the media player 403 to request the transcoded audio and video clip.

[0052] In this embodiment, the content provider 402 is responsible for uploading the original audio and video content (such as Figure 4 As shown in ①), the description of the original audio content can refer to the above embodiment, which will not be repeated here. The cloud storage node 401 stores the original audio and video content, and the original audio and video content has a first encoding format.

[0053] When the cloud storage node 401 receives the original audio and video content, the media manager 4011 can slice the original audio and video content (such as Figure 4 As shown in ② in the figure). In this process, the original audio and video content can be sliced ​​according to time periods, and the length of each slice can be several seconds or tens of seconds, and each slice is called an audio and video segment; or it can also be sliced ​​according to the content theme expressed by the original audio and video content, and continuous audio and video frames expressing the same content theme are divided into an audio and video segment, and there is no restriction on this. Optionally, text information can be extracted from the audio information contained in each audio frame and the video information contained in each video frame in the original audio and video content, and semantic segmentation can be performed based on the extracted text information to determine the continuous audio and video frames that express the same content theme. Further optionally, in the process of segmenting audio and video segments, a large language model can be combined, such as a semantic recognition model, or a duration statistical model.

[0054] In this embodiment, the original audio and video content may support different picture quality standards, such as Blu-ray, standard definition, high definition and other picture quality standards, and different picture quality standards correspond to different transcoding parameters. According to the different picture quality standards supported by the original audio and video content, different playlists may be generated for the original audio and video content, such as Blu-ray, standard definition, high definition and other picture quality standards corresponding to different playlists (such as Figure 4③). The picture quality standard has a corresponding relationship with the playlist, one picture quality standard corresponds to one playlist, one playlist corresponds to a set of transcoding parameters, each playlist includes identification information of each original audio and video segment segmented from the original audio and video content and its corresponding address information on the cloud storage node 401, and the storage location corresponding to each address information is used to store the transcoded audio and video segment obtained by transcoding the original audio and video segment.

[0055] In this embodiment, the cloud storage node 401 can provide the generated playlist to the media player 403 at the user end. The media player 403 at the user end obtains the audio and video clips to be played from the cloud storage node 401 on demand based on the playlist and plays the audio and video clips. The media player 403 can be various software programs with audio and video content playback functions, for example, various applications, small videos, social software, plug-ins or functional modules with audio and video content playback functions. In actual applications, the user opens the media player 403, and the media player 403 provides a playback interface, which may include at least one picture quality standard for the user to select, such as high definition, standard definition, blue-ray, etc., and may also include a playlist adapted to each picture quality standard. Taking a TV series as an example, each episode from episode 1 to episode N can be used as the identification information of each audio and video clip in the playlist, and the address information corresponding to each audio and video clip may not be displayed. The user can select the picture quality standard as needed, or select the episode to be played as needed, and even during the audio and video playback process, the audio and video clip to be skipped can be selected through the playback progress bar.

[0056] For the media player 403, in response to various operations initiated by the user, the media player 403 sends a play request (such as Figure 4 As shown in ④ in the figure, the play request is used to request to play the transcoded audio and video clip stored in the storage location corresponding to the target address information in the target play list. Optionally, the play request carries the target address information, as well as identification information of any picture quality standard such as HD, SD, Blu-ray, etc. The identification information of the picture quality standard carried in the play request points to the target play list, and the target address information points to the transcoded audio and video clip stored in the storage location corresponding to the target address information in the target play list, rather than other transcoded audio and video clips; wherein the target play list can be any play list in at least one play list corresponding to the original audio and video content, and the target address information can be any address information in at least one address information contained in the target play list.

[0057] The media manager 4011 obtains the transcoded audio and video clips according to the play request to the target address information. If the transcoded audio and video clips are not obtained at the storage location corresponding to the target address information, the media manager 4011 performs transcoding processing as needed (such as Figure 4 ⑤). Specifically, the original audio and video clips at the storage location corresponding to the target address information are used as the audio and video content to be transcoded, and a set of transcoding parameters corresponding to the target playlist are used as the target transcoding parameters. Further, according to the transcoding parameters corresponding to the audio frames and video frames in the target transcoding parameters, the audio frames and video frames in the audio and video content to be transcoded are transcoded respectively; if a target frame fails to be transcoded during the transcoding process, a supplementary frame corresponding to the target frame is generated according to the transcoding parameters corresponding to the target frame, and the target frame includes an audio frame and / or a video frame; the supplementary frame is added as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame, so as to obtain the transcoded audio and video content corresponding to the audio and video content to be transcoded.

[0058] See also Figure 4 As shown in ⑥, when the transcoded audio and video content corresponding to the audio and video content to be transcoded is obtained, on the one hand, it is stored in the storage location corresponding to the target address information of the cloud storage node 401, so that the media player or other media players can directly obtain the transcoded audio and video content from the target address information later, without the need for secondary transcoding, which can save the computing resources of the media manager and reduce the processing burden of the media manager. On the other hand, see Figure 4 As shown in ⑦, the transcoded audio and video content is provided to the media player 403, so that the media player 403 can continuously play the transcoded audio and video content. The transcoded audio and video content includes a transcoding result frame, and the transcoding result frame at least includes a supplementary frame corresponding to the target frame. Figure 4 As shown in ⑧, when the media player receives the transcoded audio and video content, it can continuously play the transcoded audio and video content.

[0059] In an optional embodiment, a supplementary frame corresponding to the target frame is generated according to the transcoding parameters corresponding to the target frame in the target transcoding parameters, including: determining the target content data required to generate the supplementary frame, the target content data including audio content data or video content data; generating the supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame and the audio content data or video content data in the target content data. Although the embodiment of the present application does not limit the frame content of the supplementary frame, certain content data is also required to generate the supplementary frame, and the content data required to generate the supplementary frame is referred to as target content data. The target content data includes audio content data or video content data, depending on the type of the target frame; if the target frame is an audio frame, the target content data is audio content data; if the target frame is a video frame, the target content data is video content data. In the embodiment of the present application, the method for determining the target content data is not limited, and several exemplary implementations are given below.

[0060] In an exemplary embodiment, default content data is preset, and the default content data includes default content data corresponding to an audio frame type and default content data corresponding to a video frame type, and a supplementary frame is generated based on the default content data adapted to the type of a target frame. Based on this, when determining the target content data required to generate a supplementary frame, the default content data adapted to the type of the target frame can be selected as the target content data according to the type of the target frame, and the type of the target frame is divided into an audio frame and a video frame. Preferably, in the case where the type of the target frame is an audio frame, the default content data corresponding to the audio frame type is selected as the target content data, and based on this, an audio frame containing the default content data corresponding to the audio frame type is generated as a supplementary frame according to the transcoding parameters corresponding to the target frame; in the case where the type of the target frame is a video frame, the default content data corresponding to the video frame type is selected as the target content data, and based on this, a video frame containing the default content data corresponding to the video frame type is generated as a supplementary frame according to the transcoding parameters corresponding to the target frame.

[0061] In another exemplary embodiment, at least one frame content generation scheme is provided, each frame content generation scheme can generate frame content, and different frame content generation schemes generate frame content in different ways, which can be selected and used as needed. Based on this, when determining the target content data required to generate the supplementary frame, the target frame content generation scheme can be selected from at least one frame content generation scheme according to the type of the target frame, the application type to which it belongs, and at least one information in the application scenario; and the target content data is generated according to the target frame content generation scheme. After the target content data is generated according to the target frame content generation scheme, the supplementary frame corresponding to the target frame can be generated according to the transcoding parameters corresponding to the target frame and the audio content data or video content data in the target content data. Among them, the application types include but are not limited to live broadcast applications, short video applications, video applications, e-commerce applications, and advertising applications. Different application scenarios can also be divided within the same application. Taking e-commerce applications as an example, it includes big promotion scenarios, live broadcast scenarios, interactive scenarios, etc., second kill scenarios, etc. Taking video applications as an example, it includes variety shows, TV dramas, movies, web dramas, homepage scenes, etc. In this optional embodiment, the target frame generation scheme can be selected based on at least one of the information including the type of the target frame, the type of application to which it belongs, and the application scenario, which is conducive to generating frame content of the supplementary frame that is more adapted to the type of the target frame, the type of application to which it belongs, and / or the application scenario, thereby improving the quality of the video playback picture based on the supplementary frame.

[0062] The following introduces the frame content generation scheme provided by the implementation of the present application: in an optional embodiment, at least one frame content generation scheme includes an interpolation scheme and a default scheme; when the target frame content generation scheme is an interpolation scheme, the contents of several audio frames and / or video frames that are adjacent to the target frame and have been successfully transcoded can be interpolated and calculated according to the type of the target frame to obtain the target content data.

[0063] Optionally, when the target frame type is an audio frame, interpolation calculation is performed on several audio frames that are adjacent to the target frame and have been successfully transcoded, and the interpolation result is used as the target content data, wherein the number of audio frames used for the interpolation calculation is not limited; when the target frame type is a video frame, interpolation calculation is performed on several video frames that are adjacent to the target frame and have been successfully transcoded, and the interpolation result is used as the target content data, wherein the number of video frames used for the interpolation calculation is not limited.

[0064] Alternatively, optionally, when the target frame type is an audio frame, the contents of several audio frames and several video frames that are adjacent to the target frame and have been successfully transcoded can also be interpolated; wherein, the contents of several video frames can be used as a reference to correct the contents of several audio frames in the interpolation calculation, and the interpolation calculation results can be used as the target content data, or the contents of several video frames can be used as a reference to correct the results of the interpolation calculation of the contents of several audio frames, and the corrected results can be used as the target content data. Further optionally, the contents of the audio frames or the interpolation results obtained by interpolating the contents of the audio frames can be semantically matched with the contents of the video frames; if the degree of match is less than a set matching threshold, the contents of the audio frames or the interpolation results obtained by interpolating the contents of the audio frames can be corrected until the degree of match is greater than or equal to the set matching threshold.

[0065] Similarly, when the target frame type is a video frame, the contents of several audio frames and several video frames that are adjacent to the target frame and have been successfully transcoded can also be interpolated; wherein, the contents of several audio frames can be used as a reference to correct the contents of several video frames in the interpolation calculation, and the interpolation calculation results can be used as the target content data, or the contents of several audio frames can be used as a reference to correct the results of the interpolation calculation of the contents of several video frames, and the corrected results can be used as the target content data. Further optionally, the contents of the video frames or the interpolation results obtained by interpolating the contents of the video frames can be semantically matched with the contents of the audio frames; if the matching degree is less than the set matching degree threshold, the contents of the video frames or the interpolation results obtained by interpolating the contents of the video frames can be corrected until the matching degree is greater than or equal to the set matching degree threshold.

[0066] There is no limitation on the selection method of several audio frames and / or video frames that are adjacent to the target frame and successfully transcoded. In an optional implementation, several audio frames and / or video frames that are adjacent to the target frame and successfully transcoded before the target frame can be selected; in another optional implementation, several audio frames and / or video frames that are adjacent to the target frame and successfully transcoded after the target frame can be selected; in yet another optional implementation, several audio frames and / or video frames that are successfully transcoded before the target frame and several audio frames and / or video frames that are successfully transcoded after the target frame can be selected at the same time. It is explained here that if the target frame is the first frame, the contents of several audio frames and / or several video frames that are successfully transcoded after the target frame can be interpolated; if the target frame is the last frame, the contents of several audio frames and / or several video frames that are successfully transcoded before the target frame can be interpolated.

[0067] Furthermore, when the target frame content generation scheme is the default scheme, the target content data is generated according to the content description information adapted to the type of the target frame in the default scheme; the default scheme includes content description information corresponding to the audio frame and content description information corresponding to the video frame. The content description information corresponding to the video frame is used to describe various content attributes of the video frame, including but not limited to: the color value and brightness value of the video content. The content description information corresponding to the audio frame is used to describe various content attributes of the audio frame, including but not limited to: the value, volume, and pitch of the sound signal. Based on the content description information corresponding to the video frame, a video frame with fixed content can be generated as a supplementary frame; based on the content description information corresponding to the audio frame, an audio frame with fixed content can be generated as a supplementary frame.

[0068] In an optional embodiment, when the target content data is the default content data or is generated according to the default scheme, generating a supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame and the audio content data or the video content data in the target content data includes: when the target frame is an audio frame, generating a silent frame as a supplementary frame according to the transcoding parameters corresponding to the target frame and the target content data, wherein the silent frame refers to an audio frame in which the energy (or amplitude) of the audio signal within a period of time is lower than a certain threshold, and may be an audio frame with a volume of zero or close to zero. Using the silent frame as a supplementary frame can avoid abrupt audio switching and bring a bad experience to the user; when the target frame is a video frame, generating a black frame as a supplementary frame according to the transcoding parameters corresponding to the target frame and the target content data, wherein the black frame is a frame of a completely black screen, and its RGB values ​​are all zero. Using the black frame as a supplementary frame can avoid abrupt video switching, provide users with time to adapt, and reduce visual fatigue caused by fast screen switching. Optionally, when the target frame is a video frame, a white frame is generated according to the transcoding parameters corresponding to the target frame and the target content data, wherein the white frame refers to a frame of a full white screen, and its RGB values ​​are all 255.

[0069] In one example, if Figure 5As shown, an original video sequence includes multiple video frames, among which the 2nd and 3rd frames fail to be transcoded, and the 1st, 4th and 5th frames are transcoded successfully. The 2nd and 3rd frames are taken as target frames, and black frames are generated as supplementary frames according to the transcoding parameters and target content data corresponding to the target frames. The supplementary frames are added as transcoding result frames of the target frames to the transcoding positions corresponding to the target frames to obtain the transcoded video sequence corresponding to the original video sequence. At this time, the transcoded video sequence includes the transcoding result frames corresponding to the 1st, 4th and 5th frames, and two black frames corresponding to the 2nd and 3rd frames between the 1st and 4th frames, and contains a total of 5 video frames, and each frame position has frame content. Based on this, when the user requests to play the transcoded video sequence, the media player can obtain the transcoded audio and video content containing 5 frames corresponding to the video sequence and play it. During the playback process, since there is frame content at each frame position, it can be played continuously without terminating the playback due to the failure of transcoding the 2nd and 3rd frames, or always playing the 1st frame. The user does not need to manually drag the progress bar to skip the 2nd and 3rd frames, which can ensure the continuity of the playback and improve the user's playback experience. At the same time, using two black frames as the transcoding result frames corresponding to the 2nd and 3rd frames can also avoid abrupt video switching, provide users with time to adapt, and reduce visual fatigue caused by fast screen switching.

[0070] In the above embodiment, an interpolation scheme and a default scheme are introduced. On this basis, when selecting a target frame content generation scheme from at least one frame content generation scheme according to at least one information of the type of the target frame, the type of application to which the audio and video content to be transcoded belongs, and the application scenario, when the target frame is a video frame, belongs to an audio and video application, or belongs to a specific application scenario, an interpolation scheme can be selected as the target frame content generation scheme; when the target frame is an audio frame, does not belong to an audio and video application, or does not belong to a specific application scenario, a default scheme can be selected as the target frame content generation scheme. The specific application scenario can be flexibly set according to the application requirements, and there is no limitation on this.

[0071] Further optionally, an implementation method of adding the supplementary frame as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame includes: the target frame has a timestamp, and the timestamp of the target frame is added to the supplementary frame; according to the timestamp, the supplementary frame is added as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame in the transcoded audio and video clip, so as to obtain the transcoded audio and video content corresponding to the audio and video content to be transcoded.

[0072] Full offline transcoding refers to the process of transcoding the original audio and video content uploaded by the content provider in an offline manner. The media player cannot play it before the transcoding is completed. Figure 6A structural diagram of an audio and video transcoding processing system provided for another exemplary embodiment of the present application. The system 600 includes: a cloud storage node 601, a content provider 602, a media player 603 at the user end, and a media manager 6011. The cloud storage node 601 is responsible for providing storage services for audio and video content to be transcoded and after transcoding; the media player 603 runs on the terminal device at the user end and is responsible for requesting the transcoded audio and video content from the cloud storage node 601 for playback; the media manager 6011 is responsible for offline transcoding the audio and video content to be transcoded, providing a basis for the media player 603 to request the transcoded audio and video content.

[0073] In this embodiment, the content provider is responsible for uploading the original audio and video content (such as Figure 6 ① as shown); when the cloud storage node 601 receives the original audio and video content, the media manager 6011 performs full transcoding processing on the entire original audio and video content as the audio and video content to be transcoded (as shown in FIG. Figure 6 As shown in ②, on the one hand, a playlist is generated (such as Figure 6 (as shown in ③).

[0074] During the transcoding process, the media manager 6011 obtains the original audio and video content and at least one set of transcoding parameters corresponding thereto; uses the original audio and video content as the audio and video content to be transcoded, and uses at least one set of transcoding parameters as target transcoding parameters, and transcodes the audio frames and video frames in the audio and video content to be transcoded according to the transcoding parameters corresponding to the audio frames and video frames in the target transcoding parameters; during the transcoding process, if a target frame fails to be transcoded, a supplementary frame corresponding to the target frame is generated according to the transcoding parameters corresponding to the target frame in the target transcoding parameters, and the target frame includes an audio frame and / or a video frame; the supplementary frame is added as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame, so as to obtain the transcoded audio and video content corresponding to the audio and video content to be transcoded. Further, the transcoded audio and video content is stored in a cloud storage node (such as Figure 6 (as shown in ④).

[0075] Regarding the method of generating a playlist: before receiving any playback request, the media manager 6011 can generate at least one playlist based on the original audio and video content and its corresponding at least one picture quality standard, wherein the picture quality standard corresponds to the playlist, one picture quality standard corresponds to one playlist, and one playlist corresponds to a set of transcoding parameters.

[0076] Further, in the case of obtaining at least one playlist, at least one playlist can be provided to the media player 603. In this way, the media player 603 can obtain the transcoded audio and video clips / contents from the cloud storage node 601 on demand based on the playlist and play them. Optionally, the original audio and video content can be segmented according to the type and size of the original audio and video content to obtain at least one audio and video clip, and then the original audio and video content can be fully transcoded to obtain at least one transcoded audio and video clip. In actual applications, the user opens the media player 603, and the media player 603 provides a playback interface. The playback interface may include at least one picture quality standard that can be selected by the user, such as high definition, standard definition, Blu-ray, etc., and may also include a playlist adapted to each picture quality standard. Taking a TV series as an example, each episode such as episode 1 to episode N can be used as identification information of each audio and video clip in the playlist, and the address information corresponding to each audio and video clip may not be displayed. Users can select the picture quality standard as needed, or the episodes to be played as needed. Even during the audio and video playback, they can select the audio and video segments to be skipped through the playback progress bar to request the audio and video segment corresponding to the current position of the progress bar. Alternatively, for audio and video content that has not been segmented, users can directly request to play the entire audio and video content.

[0077] The media player 603 can respond to various operations initiated by the user and send a play request to the media manager 6011. When the transcoded audio and video content is obtained offline, the media manager 6011 can receive the play request of the media player 603 at the user end (such as Figure 6 As shown in ⑤), the play request carries the target address information and the identification information of any picture quality standard such as high definition, standard definition, and blue-ray; the identification information of the picture quality standard carried in the play request points to the target play list, and the target address information points to the transcoded audio and video clips / contents stored in the storage location corresponding to the target address information in the target play list; the media manager 6011 obtains the transcoded audio and video clips / contents according to the play request to the target address information, and sends the transcoded audio and video clips / contents to the media player 603 (as shown in ⑤). Figure 6 ⑥), so that the media player 603 at the user end can play the transcoded audio and video clips / contents. Figure 6 As shown in ⑦, when the media player receives the transcoded audio and video clips / contents, it can continuously play the transcoded audio and video clips / contents.

[0078] In an optional embodiment, a supplementary frame corresponding to the target frame is generated according to the transcoding parameters corresponding to the target frame in the target transcoding parameters, including: determining the target content data required to generate the supplementary frame, the target content data including audio content data or video content data; generating the supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame and the audio content data or video content data in the target content data. Although the embodiment of the present application does not limit the frame content of the supplementary frame, certain content data is also required to generate the supplementary frame, and the content data required to generate the supplementary frame is referred to as target content data. The target content data includes audio content data or video content data, depending on the type of the target frame; if the target frame is an audio frame, the target content data is audio content data; if the target frame is a video frame, the target content data is video content data. In the embodiment of the present application, the method for determining the target content data is not limited, and several exemplary implementations are given below.

[0079] In an optional embodiment, default content data is preset, and the default content data includes default content data corresponding to the audio frame type and default content data corresponding to the video frame type, and the supplementary frame is generated based on the default content data adapted to the type of the target frame. Based on this, when determining the target content data required to generate the supplementary frame, the default content data adapted to the type of the target frame can be selected as the target content data according to the type of the target frame. The types of the target frame are divided into audio frames and video frames. For how to select the default content data adapted to the type of the target frame as the target content data according to the type of the target frame, reference can be made to the aforementioned embodiment, and no further description is given here.

[0080] In another optional embodiment, at least one frame content generation scheme is provided, each frame content generation scheme can generate frame content, and different frame content generation schemes generate frame content in different ways, which can be selected and used as needed. Based on this, when determining the target content data required to generate the supplementary frame, a target frame content generation scheme can be selected from at least one frame content generation scheme according to at least one information of the type of the target frame, the type of application to which it belongs, and the application scenario; and the target content data is generated according to the target frame content generation scheme.

[0081] In an optional embodiment, at least one frame content generation scheme includes an interpolation scheme and a default scheme; when the target frame content generation scheme is an interpolation scheme, the contents of the audio frames and / or video frames adjacent to and transcoded successfully before and after the target frame are interpolated according to the type of the target frame to obtain the target content data. The contents of the interpolation calculation can be referred to the aforementioned embodiment and will not be described in detail here.

[0082] Further, when the target frame content generation scheme is the default scheme, the target content data is generated according to the content description information adapted to the type of the target frame in the default scheme; the default scheme includes content description information corresponding to the audio frame and content description information corresponding to the video frame. For details of the content description information, please refer to the aforementioned embodiment and will not be repeated here.

[0083] In an optional embodiment, when the target content data is the default content data or is generated according to the default scheme, a supplementary frame corresponding to the target frame is generated according to the transcoding parameters and the target content data corresponding to the target frame, including: when the target frame is an audio frame, a silent frame is generated as a supplementary frame according to the transcoding parameters and the target content data corresponding to the target frame, wherein the silent frame refers to an audio frame in which the energy (or amplitude) of the audio signal within a period of time is lower than a certain threshold, and may be an audio frame with a volume of zero or close to zero. Using the silent frame as a supplementary frame can avoid abrupt audio switching and bring a bad experience to the user; when the target frame is a video frame, a black frame is generated as a supplementary frame according to the transcoding parameters and the target content data corresponding to the target frame, wherein the black frame is a frame of a completely black screen with an RGB value of zero. Using the black frame as a supplementary frame can avoid abrupt video switching, provide users with time to adapt, and reduce visual fatigue caused by fast screen switching.

[0084] In the above embodiment, an interpolation scheme and a default scheme are introduced. On this basis, when selecting a target frame content generation scheme from at least one frame content generation scheme according to at least one information of the type of the target frame, the type of application to which the audio and video content to be transcoded belongs, and the application scenario, when the target frame is a video frame, belongs to an audio and video application, or belongs to a specific application scenario, an interpolation scheme can be selected as the target frame content generation scheme; when the target frame is an audio frame, does not belong to an audio and video application, or does not belong to a specific application scenario, a default scheme can be selected as the target frame content generation scheme. The specific application scenario can be flexibly set according to the application requirements, and there is no limitation on this.

[0085] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0086] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 201 to 203 can be device A; for another example, the execution subject of steps 201 and 202 can be device A, and the execution subject of step 203 can be device B; and so on.

[0087] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 201, 202, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0088] Figure 7 A structural diagram of a media manager provided by another exemplary embodiment of the present application is shown in FIG. Figure 7 As shown, the media manager includes: a memory 74 and a processor 75.

[0089] The memory 74 is used to store computer programs and can be configured to store various other data to support operations on the media manager. Examples of these data include instructions, messages, pictures, videos, etc. for any application or method operating on the media manager.

[0090] The processor 75 is coupled to the memory 74 and is used to execute the computer program in the memory 74, so as to: obtain the audio and video content to be transcoded and its corresponding target transcoding parameters; transcode the audio and video content to be transcoded according to the target transcoding parameters; determine the target frame in which transcoding fails in the audio and video content to be transcoded, and generate a supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame; add the supplementary frame as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame to obtain the transcoded audio and video content corresponding to the audio and video content to be transcoded.

[0091] In an optional embodiment, when the processor 75 obtains the audio and video content to be transcoded and its corresponding target transcoding parameters, it is specifically used to: receive a play request sent by the player, the play request is used to request to play the transcoded audio and video clip stored in the storage location corresponding to the target address information in the target playlist; wherein the target playlist is any playlist in at least one playlist corresponding to the original audio and video content, one playlist corresponds to a set of transcoding parameters, and one playlist includes at least one address information including the target address information, and the storage location corresponding to the at least one address information is used to store the transcoded audio and video clip corresponding to at least one original audio and video clip cut from the original audio and video content; if the transcoded audio and video clip is not obtained at the storage location corresponding to the target address information, the original audio and video clip corresponding to the target address information is used as the audio and video content to be transcoded, and a set of transcoding parameters corresponding to the target playlist is used as the target transcoding parameters.

[0092] In an optional embodiment, the processor 75 is also used to: before receiving the playback request, generate the at least one play list according to the original audio and video content and its corresponding at least one picture quality standard, wherein one picture quality standard corresponds to a set of transcoding parameters; and after obtaining the transcoded audio and video content, store the transcoded audio and video content in a storage location corresponding to the target address information, and send the transcoded audio and video content to the player so that the player can play the transcoded audio and video content.

[0093] In an optional embodiment, when the processor 75 obtains the audio and video content to be transcoded and its corresponding target transcoding parameters, it is specifically used to: obtain the original audio and video content and its corresponding at least one set of transcoding parameters; before receiving any playback request, use the original audio and video content as the audio and video content to be transcoded, and use the at least one set of transcoding parameters as the target transcoding parameters, respectively, and one picture quality standard corresponds to a set of transcoding parameters.

[0094] In an optional embodiment, when the processor 75 generates a supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame, it is specifically used to: determine the target content data required to generate the supplementary frame; and generate the supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame and the target content data.

[0095] In an optional embodiment, the audio and video content to be transcoded includes audio frames and video frames, and the target frame is an audio frame or a video frame. Then, when the processor 75 determines the target content data required to generate the supplementary frame, it is specifically used to: select default content data adapted to the type of the target frame as the target content data according to the type of the target frame; or select a target frame content generation scheme from at least one frame content generation scheme according to at least one piece of information including the type of the target frame, the application type to which it belongs, and the application scenario; and generate the target content data according to the target frame content generation scheme.

[0096] In an optional embodiment, when the processor 75 generates the target content data according to the target frame content generation scheme, it is specifically used to: when the target frame content generation scheme is an interpolation scheme, according to the type of the target frame, interpolate the contents of the audio frames and / or video frames that are adjacent to the target frame and have been successfully transcoded to obtain the target content data; when the target frame content generation scheme is a default scheme, generate the target content data according to the content description information adapted to the type of the target frame in the default scheme.

[0097] In an optional embodiment, when the processor 75 generates a supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame and the target content data, it is specifically used to: when the target frame is an audio frame, generate a silent frame as the supplementary frame according to the transcoding parameters corresponding to the target frame and the target content data; when the target frame is a video frame, generate a black frame as the supplementary frame according to the transcoding parameters corresponding to the target frame and the target content data.

[0098] In an optional embodiment, when the processor 75 adds the supplementary frame as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame, it is specifically used to: the target frame has a timestamp, and the timestamp of the target frame is added to the supplementary frame; according to the timestamp, the supplementary frame is added as the transcoding result frame of the target frame to the transcoding position corresponding to the target frame in the transcoded audio and video clip.

[0099] Further, if Figure 7 As shown, the media manager also includes: a communication component 76, a display 77, a power component 78, an audio component 79 and other components. Figure 7 The components are shown in the figure only for reference. This does not mean that the media manager only includes Figure 7 In addition, Figure 7The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the media manager. The media manager of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the media manager of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 7 If the media manager of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 7 Components within the dashed box.

[0100] The present application also provides an electronic device, the implementation structure of which is as follows: Figure 8 As shown, it includes: a memory 84 and a processor 85.

[0101] The memory 84 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0102] The processor 85 is coupled to the memory 84 and is used to execute the computer program in the memory 84, so as to: send a play request to the media manager to request the media manager to transcode the audio and video content, wherein the transcoded audio and video content is obtained by transcoding the audio and video content to be transcoded; receive the transcoded audio and video content sent by the media manager, wherein the transcoded audio and video content includes a transcoding result frame, and the transcoding result frame at least includes a supplementary frame corresponding to a target frame determined to have failed to be transcoded in the audio and video content to be transcoded; and play the transcoding result frame in the transcoded audio and video content.

[0103] Further, if Figure 8 As shown, the electronic device also includes: a communication component 86, a display 87, a power component 88, an audio component 89 and other components. Figure 8 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 8 In addition, the electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT device, or a server device such as a conventional server, a cloud server or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 8 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 8The components in the dotted box can provide a web interface for playing audio and video content, and display the web interface on the screen of the user terminal device to play the audio and video content to the user.

[0104] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor is enabled to implement the steps in the above method.

[0105] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0106] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0107] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0108] The display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0109] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0110] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program codes.

[0112] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0116] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0117] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0118] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0119] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for audio and video transcoding, characterized in that: The method comprises: Obtain the audio and video content to be transcoded and its corresponding target transcoding parameters; According to the target transcoding parameters, the audio and video content to be transcoded is transcoded, wherein the transcoding process includes multiple steps, including: decapsulation, decoding, preprocessing, encoding and encapsulation; Determine a target frame in the audio and video content to be transcoded that fails to be transcoded, and generate a supplementary frame corresponding to the target frame according to a transcoding parameter in the target transcoding parameter that corresponds to the target frame, wherein the target frame is an audio frame or a video frame in any of the multiple steps where transcoding fails. If the step in which the target frame is located is not the last step in the multiple steps, information indicating that transcoding of the target frame fails is sequentially transmitted to a subsequent step of any of the steps, so that the operation of generating the supplementary frame is completed in the subsequent step; The supplementary frame is added as a transcoding result frame of the target frame to a transcoding position corresponding to the target frame, so as to obtain transcoded audio and video content corresponding to the audio and video content to be transcoded.

2. The method according to claim 1, characterized in that Generating a supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame, including: determining target content data required to generate the supplementary frame; A supplementary frame corresponding to the target frame is generated according to the transcoding parameters corresponding to the target frame and the target content data.

3. The method according to claim 2, characterized in that The audio and video content to be transcoded includes an audio frame and a video frame, and the target frame is an audio frame or a video frame, then determining the target content data required to generate the supplementary frame includes: According to the type of the target frame, selecting default content data adapted to the type of the target frame as the target content data; or According to at least one of the type of the target frame, the type of application to which it belongs, and the application scenario, a target frame content generation scheme is selected from at least one frame content generation scheme; and according to the target frame content generation scheme, the target content data is generated.

4. The method according to claim 3, characterized in that Generating the target content data according to the target frame content generation scheme includes: In the case where the target frame content generation scheme is an interpolation scheme, interpolation calculation is performed on the contents of audio frames and / or video frames adjacent to the target frame and successfully transcoded according to the type of the target frame to obtain the target content data; In the case where the target frame content generation scheme is a default scheme, the target content data is generated according to content description information in the default scheme adapted to the type of the target frame.

5. The method according to any one of claims 2 to 4, characterized in that Generating a supplementary frame corresponding to the target frame according to the transcoding parameters corresponding to the target frame and the target content data, including: In the case where the target frame is an audio frame, generating a silent frame as the supplementary frame according to the transcoding parameters corresponding to the target frame and the target content data; In the case where the target frame is a video frame, a black frame is generated according to the transcoding parameters corresponding to the target frame and the target content data as the supplementary frame.

6. The method according to claim 1, characterized in that Get the audio and video content to be transcoded and its corresponding target transcoding parameters, including: Receive a play request sent by a player, wherein the play request is used to request to play the transcoded audio and video clip stored in a storage location corresponding to the target address information in the target playlist; The target playlist is any playlist in at least one playlist corresponding to the original audio and video content, one playlist corresponds to a set of transcoding parameters, and one playlist includes at least one address information including the target address information, and the storage location corresponding to the at least one address information is used to store the transcoded audio and video segment corresponding to at least one original audio and video segment cut from the original audio and video content; If the transcoded audio and video clip is not obtained at the storage location corresponding to the target address information, the original audio and video clip corresponding to the target address information is used as the audio and video content to be transcoded, and a set of transcoding parameters corresponding to the target playlist is used as the target transcoding parameters.

7. The method according to claim 6, characterized in that The method further comprises: Before receiving the playback request, generating the at least one playlist according to the original audio and video content and at least one picture quality standard corresponding thereto, wherein one picture quality standard corresponds to a set of transcoding parameters; and After obtaining the transcoded audio and video content, the transcoded audio and video content is stored in a storage location corresponding to the target address information, and the transcoded audio and video content is sent to the player so that the player can play the transcoded audio and video content.

8. The method according to claim 1, characterized in that Get the audio and video content to be transcoded and its corresponding target transcoding parameters, including: Obtain original audio and video content and its corresponding at least one set of transcoding parameters; before receiving any playback request, use the original audio and video content as the audio and video content to be transcoded, and use the at least one set of transcoding parameters as the target transcoding parameters, one picture quality standard corresponds to one set of transcoding parameters.

9. The method according to any one of claims 1-4, 6-8, characterized in that: Adding the supplementary frame as a transcoding result frame of the target frame to a transcoding position corresponding to the target frame includes: The target frame has a timestamp, and the timestamp of the target frame is added to the supplementary frame; According to the timestamp, the supplementary frame is added as a transcoding result frame of the target frame to a transcoding position corresponding to the target frame in the transcoded audio and video segment.

10. An audio and video playback method, characterized in that: include: Sending a play request to a media manager to request the media manager to transcode the audio and video content, wherein the transcoded audio and video content is obtained by transcoding the audio and video content to be transcoded, and the transcoding process includes multiple steps, including: decapsulation, decoding, preprocessing, encoding and encapsulation; Receive the transcoded audio and video content sent by the media manager, the transcoded audio and video content includes a transcoding result frame, the transcoding result frame at least includes a supplementary frame corresponding to a target frame determined to have failed transcoding in the audio and video content to be transcoded, the target frame is an audio frame or a video frame in which transcoding fails in any of the multiple steps, and when the step where the target frame is located is not the last step of the multiple steps, pass information indicating that the transcoding of the target frame fails to the subsequent steps of any of the steps in sequence, so that the operation of generating the supplementary frame is completed in the subsequent step; Play the transcoded result frames in the transcoded audio and video content.

11. A media manager, characterized in that: include: Memory and processor; The memory is used to store a computer program, and the processor is coupled to the memory and is used to execute the computer program in the memory to implement the steps in the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program / instruction, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that include: A computer program / instruction, when executed by a processor, causes the processor to implement the steps of the method according to any one of claims 1 to 10.

Images