Multimedia data transmission control method and device, equipment and storage medium

Abstract

The invention discloses a multimedia data transmission control method and device, equipment and a storage medium. The method comprises the following steps: receiving a first media code stream and a second media code stream which are formed by encoding the same original media file; sending the first media code stream to a receiving end; caching the second media code stream; when the redundant bandwidth exists in the transmission network, the cached second media code stream is sent to the receiving end; wherein the process of sending the second media code stream comprises the following steps: when an interested time period sent by a receiving end is received, analyzing an audio corresponding to the interested time period to obtain an interested topic; analyzing an audio in a second media code stream which is not sent yet, searching for an audio clip of which the association degree with the interested topic is within a preset threshold range, and obtaining a corresponding matching time period; and preferentially sending the second media code stream corresponding to the matching time period. According to the invention, a user can watch a playback video with relatively high quality during playback, and the watching experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia data transmission, and more specifically, to a multimedia data transmission control method, device, equipment and storage medium. Background Art

[0002] With the continuous development of camera technology, network broadband technology, video decompression technology and storage technology, remote video systems such as video conferencing and distance learning are widely used in various occasions. In practice, in order to save the real-time transmitted media stream, it is usually necessary to record the media stream. For example, online video recording can record and save video data of video sources such as video conferencing and digital television that are watched in real time and on demand on the Internet for subsequent viewing. The above-mentioned online videos generally need to be encoded and compressed before transmission. The encoding quality of video images, such as clarity and smoothness, is generally affected by network bandwidth.

[0003] In the existing technical solution, the recording and broadcasting of online videos is performed by directly saving the real-time video stream as the content of the recording and playback. For example: in a video conference, terminal A and terminal B conduct a multi-screen conference through a media server (MCU: Multipoint Control Unit). When terminal B wants to save the meeting record, it will receive the multi-screen video stream from the MCU during the meeting. On the one hand, it will play the received multi-screen video stream in real time, and on the other hand, it will directly record the multi-screen video stream and save it in the hard disk for playback.

[0004] In the process of recording and broadcasting the above-mentioned online videos, if the video is transmitted with good quality and high definition, it will occupy excessive bandwidth and cannot guarantee the smoothness of the real-time playback at the receiving end. If the video is transmitted with poor quality, although the smoothness of the real-time playback can be guaranteed, the poor quality video is not suitable for playback.

[0005] In order to solve the above problems, the patent application with publication number CN102111644A discloses a media transmission control method, in which two kinds of videos, high quality and low quality, are transmitted simultaneously, and the low quality video is transmitted first to ensure the smoothness of real-time playback. When the network has redundant bandwidth, the high quality video is transmitted again for storage and playback. However, in this method, high-quality videos are transmitted in the order of playback, which will result in the later the playback order, the later the receiving end receives it. If the user wants to watch the video later in the order, he may be forced to watch the poor quality video because it is still being received, which greatly reduces the user experience. Summary of the invention

[0006] In order to overcome at least one of the defects of the prior art described above, the present invention provides a multimedia data transmission control method, device, equipment and storage medium. The technical solution adopted by the present invention is as follows.

[0007] In a first aspect, the present invention provides a multimedia data transmission control method, comprising:

[0008] Receiving a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream;

[0009] Sending the first media code stream to a receiving end;

[0010] caching the second media code stream;

[0011] When redundant bandwidth exists in the transmission network, sending the cached second media code stream to the receiving end;

[0012] The process of sending the cached second media code stream to the receiving end includes:

[0013] When receiving the time period of interest sent by the receiving end, analyzing the audio corresponding to the time period of interest to obtain the topic of interest;

[0014] Analyze the audio in the second media stream that has not yet been sent, find the audio segment whose relevance to the topic of interest is within a preset threshold range, and obtain the corresponding matching time period;

[0015] The second media code stream corresponding to the matching time period is sent preferentially.

[0016] In one implementation, before analyzing the audio corresponding to the time period of interest to obtain the topic of interest, the method further includes:

[0017] Determine whether the number of unsent second media code streams is gradually increasing;

[0018] When the number of unsent second media code streams increases gradually, the process of analyzing the audio corresponding to the interested time period to obtain the interested topic is performed.

[0019] In one embodiment, the process of analyzing the audio in the second media stream that has not yet been sent, searching for an audio segment whose relevance to the topic of interest is within a preset threshold range, and obtaining a corresponding matching time period includes:

[0020] Analyze the audio in each GOP in the second media code stream that has not yet been sent to obtain a topic analysis result of each GOP;

[0021] According to the topic analysis results of each GOP, searching for a GOP whose relevance to the topic of interest is greater than a preset threshold;

[0022] The time periods corresponding to all GOPs whose relevance to the topic of interest is greater than a preset threshold are determined as matching time periods.

[0023] In one implementation, the process of preferentially sending the second media code stream corresponding to the matching time period includes:

[0024] In the second media code stream corresponding to the matching time period, the corresponding GOPs are sent in order according to the relevance of the topic analysis results of the respective GOPs to the topic of interest.

[0025] In one implementation, the process of preferentially sending the second media code stream corresponding to the matching time period includes:

[0026] Receive the current playback time point of the receiving end;

[0027] Calculate, according to the current playback time point, a predicted playback time point at which the second media code stream corresponding to each matching time period is played at the receiving end;

[0028] Get network status;

[0029] Calculate, according to the network condition, a predicted receiving time point at which the second media code stream corresponding to each matching time period arrives at the receiving end;

[0030] In the second media code streams corresponding to the matching time period, the second media code streams whose predicted playback time point is later than the predicted reception time point in the second media code streams corresponding to the matching time period are preferentially sent.

[0031] In one implementation, the process of analyzing the audio corresponding to the time period of interest to obtain the topic of interest includes:

[0032] When no topic of interest can be obtained by analyzing the audio corresponding to the time period of interest, extending the length of the time period of interest to obtain an extended time period of interest;

[0033] The audio corresponding to the extended time period of interest is analyzed to obtain topics of interest.

[0034] In one implementation, the process of sending the cached second media code stream to the receiving end further includes:

[0035] When there is no matching time period or there is no corresponding second media code stream in the matching time period, the cached second media code stream is sent to the receiving end in sequence.

[0036] In a second aspect, the present invention provides a multimedia data transmission control method, comprising:

[0037] A sending end receives a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream;

[0038] The sending end sends the first media code stream to the receiving end;

[0039] The sending end caches the second media code stream;

[0040] When redundant bandwidth exists in the transmission network, the sending end sends the cached second media code stream to the receiving end;

[0041] The receiving end receives and stores the received first media code stream and the second media code stream;

[0042] When the receiving end receives the playback instruction, it determines whether the current continuous playback duration exceeds the threshold duration. If the current continuous playback duration exceeds the threshold duration, it obtains the time period of interest based on the current playback time point and sends it to the sending end;

[0043] The process of sending the cached second media code stream from the sending end to the receiving end includes:

[0044] When the sending end receives the time period of interest sent by the receiving end, the audio corresponding to the time period of interest is analyzed to obtain the topic of interest;

[0045] The sending end analyzes the audio in the second media code stream that has not yet been sent, searches for an audio segment whose relevance to the topic of interest is within a preset threshold range, and obtains a corresponding matching time period;

[0046] The sending end preferentially sends the second media code stream corresponding to the matching time period.

[0047] In a third aspect, the present invention provides a sending end multimedia data transmission control device, comprising:

[0048] A receiving module, configured to receive a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream;

[0049] A first sending module, used for sending the first media code stream to a receiving end;

[0050] A cache module, used for caching the second media code stream;

[0051] A second sending module, used for sending the cached second media code stream to the receiving end when there is redundant bandwidth in the transmission network;

[0052] The operation of the second sending module in the process of sending the cached second media code stream to the receiving end includes:

[0053] When receiving the time period of interest sent by the receiving end, analyzing the audio corresponding to the time period of interest to obtain the topic of interest;

[0054] Analyze the audio in the second media stream that has not yet been sent, find the audio segment whose relevance to the topic of interest is within a preset threshold range, and obtain the corresponding matching time period;

[0055] The second media code stream corresponding to the matching time period is sent preferentially.

[0056] In a fourth aspect, the present invention provides a sending end multimedia data transmission control system, comprising: a sending end and a receiving end;

[0057] Wherein, the transmitting end includes the transmitting end multimedia data transmission control device mentioned above, and the receiving end includes the receiving end multimedia data transmission control device;

[0058] The receiving end multimedia data transmission control device comprises:

[0059] A receiving module, used for receiving and storing the first media code stream and the second media code stream;

[0060] The processing module is used to determine whether the current continuous playback duration exceeds the threshold duration when receiving the playback instruction. If the current continuous playback duration exceeds the threshold duration, the interested time period is obtained according to the current playback time point and sent to the sending end.

[0061] In a fifth aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned methods when executing the program.

[0062] In a sixth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method of any of the above-mentioned embodiments is implemented.

[0063] In the present invention, while ensuring that the first media stream with low quality can be sent smoothly, the cached second media stream with high quality is selected and prioritized. The receiving end obtains the user's interested time period by observing the user's continuous playback duration. The sending end analyzes and compares the audio data of the user's interested time period captured by the receiving end with the audio of the second media stream that has not been sent, finds the part with a high matching degree between the two in the second media stream, and sends the content of this part preferentially. The present invention enables users to watch high-quality and highly relevant playback videos during playback, improving the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the overall process of an implementation method of Example 1 of the present invention.

[0065] Figure 2 It is a schematic diagram of the overall flow of an implementation of the sending process of the first embodiment of the present invention.

[0066] Figure 3 It is a schematic diagram of the overall process of an implementation method of the second embodiment of the present invention.

[0067] Figure 4 It is a schematic diagram of the overall structure of the third embodiment of the present invention. DETAILED DESCRIPTION

[0068] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0069] It should be noted that the terms "first\second\..." involved in the embodiments of the present invention are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\..." can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\..." can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0070] Embodiment 1

[0071] See also Figure 1 , Figure 11 is a flowchart of a multimedia data transmission control method provided in Embodiment 1 of the present invention, the method comprising step S110, step S120, step S130 and step S140. It should be noted that step S110, step S120, step S130 and step S140 are merely reference numerals for the purpose of clearly explaining the embodiment and the appendix. Figure 1 The corresponding relationship does not limit the order of the steps in this embodiment.

[0072] Step S110, receiving a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream;

[0073] Step S120, sending the first media code stream to the receiving end;

[0074] Step S130, caching the second media code stream;

[0075] Step S140: When redundant bandwidth exists in the transmission network, the cached second media code stream is sent to the receiving end.

[0076] like Figure 2 As shown, the process of sending the cached second media code stream to the receiving end in step S140 includes: step S210, step S220, and step S230.

[0077] Step S210, when receiving the time period of interest sent by the receiving end, analyzing the audio corresponding to the time period of interest to obtain the topic of interest;

[0078] Step S220, analyzing the audio in the second media stream that has not yet been sent, searching for an audio segment whose relevance to the topic of interest is within a preset threshold range, and obtaining a corresponding matching time period;

[0079] Step S230: preferentially sending the second media code stream corresponding to the matching time period.

[0080] This method is applicable to a sending end, such as a server or an MCU end. The original media file is encoded into a first media code stream and a second media code stream, wherein the first media code stream is a code stream of slightly poor quality that can meet the quality requirements of real-time playback, and the second media code stream is a code stream of relatively good quality that is suitable for meeting the quality requirements of recording. In step S110, the two code streams are obtained simultaneously.

[0081] In step S120, the first media code stream meets the quality requirements of real-time playback, so it is directly sent to the receiving end for real-time playback. In step S130, during the transmission of the first media stream, the second media code stream is first cached at the sending end and is not sent immediately, so that the bandwidth of the first media stream can be guaranteed, effectively ensuring the real-time and smooth transmission of the first media code stream.

[0082] In the process of implementing step S104, after the second media code stream is cached, it is sent when redundant bandwidth exists in the transmission network. Since the quality of the second media code stream is higher than that of the first media code stream, not only can the redundant bandwidth in the transmission network be fully utilized to send the second media code stream with improved media code stream recording quality, but the second media code stream can also be used as content for recording and playback, thereby improving the quality of the recorded media code stream.

[0083] In step S140, if the interested time period sent by the receiving end is not received, the cached second media code stream is sent to the receiving end in order. When the interested time period sent by the receiving end is received, the cached second media code stream is prioritized and sent in order of priority.

[0084] When users replay, they usually browse at different time points to see if there is content they want to watch. When they encounter content they want to watch, they will not click other time points, and start watching continuously from this time point, or users always browse within certain time periods, etc. The receiving end can preset some models that derive the user's interested time period based on the user's playback process as mentioned above, derive the interested time period through the model, and send the interested time period to the sending end.

[0085] In step S210, the audio corresponding to the time period of interest is analyzed to obtain the topic of interest. The receiving end directly provides the time period of interest, and the sending end (such as the server end) searches for the corresponding voice segment through the time period of interest, and then converts the voice segment into a text sentence. After obtaining the text sentence, the text sentence can be directly used as the topic of interest, or the text sentence can be sent to a trained text model to obtain the topic of interest that can represent the text sentence.

[0086] It should be noted here that, since the contents of the audio data of the first media code stream, the second media code stream and the original media file are the same, the process of analyzing the audio can be to analyze the audio data of any of the above data to obtain the topic of interest.

[0087] In step S220, the second media code stream that has not been sent is subjected to an action similar to step S210, and the audio therein is converted into a text sentence. After the text sentence is obtained, it is sent to a trained text matching model, matched with the topic of interest, and a relevance score with the topic of interest is obtained. After the relevance score is obtained, it is compared with a preset threshold range, and the text sentences within the preset threshold range are screened out, and the time period corresponding to these text sentences in the second media code stream that has not been sent is inferred, and this time period is the matching time period. In step S230, the second media code streams corresponding to these matching time periods are preferentially sent.

[0088] In this method, while ensuring that the low-quality first media stream can be sent smoothly, the cached high-quality second media stream is selected and prioritized. By analyzing and comparing the audio data of the user's interested time period captured by the receiving end with the audio of the second media stream that has not been sent, the part with a high matching degree between the two is found in the second media stream, and the content of this part is sent first. This method allows users to watch high-quality and highly relevant playback videos during playback, improving the user's viewing experience.

[0089] In one implementation, before the process of analyzing the audio corresponding to the time period of interest to obtain the topic of interest in step S210, the process further includes: step S211 and step S212.

[0090] Step S211, determining whether the number of unsent second media code streams is gradually increasing;

[0091] Step S212, when the unsent second media code streams show a trend of gradually increasing, executing the process of analyzing the audio corresponding to the interested time period to obtain the interested topic.

[0092] The unsent second media code stream generally shows three trends at the sending end: gradually decreasing, gradually increasing, and sometimes more and sometimes less. If it is gradually decreasing, it means that the current sequential sending method can effectively and quickly send the second media code stream to the receiving end, and it is not meaningful to send the content that the audience is interested in and may be played to the receiving end in advance. If it is sometimes more and sometimes less, it means that although there is a certain backlog, the second media code stream can be transmitted within a certain period of time. At this time, other methods can be used, such as converting some I frames into P frames to reduce the amount of data, so that the second media code stream can be transmitted to the receiving end more smoothly. The method of predicting and sending in advance is not meaningful. If it is gradually increasing, it means that the later the video is, the more difficult it is to send it to the receiving end in time, so at this time, the advantage of sending the content that the audience is interested in and may be played to the receiving end in advance is particularly obvious. Therefore, in this embodiment, the step of sending the content that the audience is interested in and may be played to the receiving end in advance is performed only when the unsent second media code stream shows a trend of gradually increasing.

[0093] In one implementation, the process of step S220 includes: step S221, step S222, and step S223.

[0094] Step S221, analyzing the audio in each GOP in the second media stream that has not been sent yet, and obtaining a topic analysis result of each GOP;

[0095] Step S222, searching for a GOP whose relevance to the topic of interest is greater than a preset threshold according to the topic analysis results of each GOP;

[0096] Step S223: determine the time periods corresponding to all GOPs whose relevance to the topic of interest is greater than a preset threshold as matching time periods.

[0097] In this embodiment, GOP is used as the shortest time unit in the matching time period to analyze the topic of GOP. In the analysis of the topic of GOP, there may be two sentences in a GOP, and the relevance of the two sentences is different. At this time, the corresponding relevance score can be obtained through the specific proportion. For example, in a GOP, the similarity of the first half is 80%, accounting for 3 / 10 of the total GOP length, and the similarity of the second half is 20%, accounting for 7 / 10 of the total GOP length. Then the similarity score of this GOP is 80%*3 / 10+20%*7 / 10.

[0098] In step S222, all GOPs with correlation greater than a preset threshold are found, and in step S223, the combined time period of these GOPs is determined as a matching time period. In step S230, the second media code streams corresponding to these matching time periods are preferentially sent, that is, these GOPs are sent.

[0099] In one implementation, the process of step S230 includes:

[0100] In the second media code stream corresponding to the matching time period, the corresponding GOPs are sent in order according to the relevance of the topic analysis results of the respective GOPs to the topic of interest.

[0101] This implementation uses the relevance score in the previous implementation, and further sends the corresponding media streams in the order of relevance, which can better adapt to the user's viewing experience.

[0102] In one implementation, the process of step S230 includes: step S231, step S232, step S233 and step S234.

[0103] Step S231, receiving the current playing time point of the receiving end;

[0104] Step S232, calculating the predicted playback time point of the second media code stream corresponding to each matching time period at the receiving end according to the current playback time point;

[0105] Step S233, obtaining network status;

[0106] Step S234, calculating the predicted receiving time point when the second media code stream corresponding to each matching time period arrives at the receiving end according to the network status;

[0107] Step S235 : in the second media code streams corresponding to the matching time period, the second media code streams whose predicted playback time point is later than the predicted reception time point in the second media code streams corresponding to the matching time period are preferentially sent.

[0108] Assume that the user is watching the video played back at time point A. If the second media stream starting from time point A is transmitted immediately, it will take time t to play after a series of steps such as transmission and receiving end processing. At this time, the user has already watched the time point B (B = A + t), and the second media stream just transmitted becomes meaningless. Therefore, the time consumed in the transmission process and the processing process is fully considered in this embodiment, and the second media stream that can be played at the corresponding playback time is sent first.

[0109] In one implementation, the process of analyzing the audio corresponding to the time period of interest to obtain the topic of interest includes:

[0110] When no topic of interest can be obtained by analyzing the audio corresponding to the time period of interest, extending the length of the time period of interest to obtain an extended time period of interest;

[0111] The audio corresponding to the extended time period of interest is analyzed to obtain topics of interest.

[0112] It is possible that there is not much audio content in the time period of interest and it does not meet the requirements for analysis. In this case, the present embodiment extends the time period of interest, that is, extends it forward and / or backward for a certain length of time to obtain an extended time period of interest, and then analyzes the audio corresponding to the extended time period of interest.

[0113] In one implementation, the process of sending the cached second media code stream to the receiving end further includes:

[0114] When there is no matching time period or there is no corresponding second media code stream in the matching time period, the cached second media code stream is sent to the receiving end in sequence.

[0115] If the receiving end does not send the time period of interest, there is naturally no matching time period, or there is no matching time period due to other reasons. At this time, the cached second media code stream is sent in the order of the original video; in addition, when all the second media code streams in the matching time period have been sent, or there are no second media code streams to be sent in the matching time period due to other reasons, the cached second media code streams are also sent in the order of the original video.

[0116] Embodiment 2

[0117] See also Figure 3 , Figure 3 The flowchart of a multimedia data transmission control method provided in the second embodiment of the present invention includes step S310, step S320, step S330, step S340, step S350 and step S360. It should be noted that step S310, step S320, step S330, step S340, step S350 and step S360 are only reference numerals for clarifying the embodiment and the appendix. Figure 3 The corresponding relationship does not limit the order of the steps in this embodiment.

[0118] Step S310: The transmitting end receives a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream;

[0119] Step S320: the sending end sends the first media code stream to the receiving end;

[0120] Step S330: the sending end caches the second media code stream;

[0121] Step S340, when there is redundant bandwidth in the transmission network, the sending end sends the cached second media code stream to the receiving end;

[0122] Step S350: the receiving end receives and stores the received first media code stream and the second media code stream;

[0123] Step S360, when the receiving end receives the playback instruction, it determines whether the current continuous playback duration exceeds the threshold duration. If the current continuous playback duration exceeds the threshold duration, the interested time period is obtained according to the current playback time point and sent to the sending end.

[0124] The process in step S340 in which the sending end sends the cached second media code stream to the receiving end includes: step S410, step S420, and step S430.

[0125] Step S410, when the sending end receives the time period of interest sent by the receiving end, the audio corresponding to the time period of interest is analyzed to obtain the topic of interest;

[0126] Step S420, the sending end analyzes the audio in the second media stream that has not been sent, searches for an audio segment whose relevance to the topic of interest is within a preset threshold range, and obtains a corresponding matching time period;

[0127] Step S430: The sending end preferentially sends the second media code stream corresponding to the matching time period.

[0128] The method is applicable to a transmission system, which includes a transmitting end and a receiving end, wherein the working process of the transmitting end is the same as that in the first embodiment. In the method, the receiving end receives and saves the first media code stream and the second media code stream sent by the transmitting end. At the same time, the receiving end also needs to promptly feedback the time period of interest to the transmitting end, so that the transmitting end can reasonably send the second media code stream according to the time period of interest. Of course, before the receiving end feedbacks, it also needs to analyze and summarize the time period of interest according to the user's operation instructions.

[0129] Specifically, when a playback instruction is received, it means that the user is playing back. Before the user actually watches the playback, he usually browses at different time points to see if it is the content he wants to watch. When he encounters the content he wants to watch, he will not click other time points, and start watching continuously from this time point. Based on this principle, the duration of the user's continuous viewing of the playback is captured in step S360. When the duration exceeds the threshold duration, it is considered that the user has started watching the playback. At this time, the current viewing time point can be captured, and a time period can be obtained by extending forward and / or backward according to this time. This time period is the time period of interest.

[0130] In this method, while ensuring that the first media stream with low quality can be sent smoothly, the cached second media stream with high quality is selected and prioritized. The receiving end obtains the user's interested time period by observing the user's continuous playback time. The sending end analyzes and compares the audio data of the user's interested time period captured by the receiving end with the audio of the second media stream that has not been sent, finds the part with a high matching degree between the two in the second media stream, and sends the content of this part first. This method allows users to watch high-quality and highly relevant playback videos during playback, improving the user's viewing experience.

[0131] Embodiment 3

[0132] Corresponding to the method of embodiment 1, as Figure 4 As shown, the present invention further provides a sending-end multimedia data transmission control device 5, comprising: a receiving module 510, a first sending module 520, a buffer module 530 and a second sending module 540.

[0133] A receiving module 510, configured to receive a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream;

[0134] A first sending module 520, configured to send the first media code stream to a receiving end;

[0135] A cache module 530, configured to cache the second media code stream;

[0136] The second sending module 540 is configured to send the cached second media code stream to the receiving end when redundant bandwidth exists in the transmission network.

[0137] The operation of the second sending module in the process of sending the cached second media code stream to the receiving end includes:

[0138] When receiving the time period of interest sent by the receiving end, analyzing the audio corresponding to the time period of interest to obtain the topic of interest;

[0139] Analyze the audio in the second media stream that has not yet been sent, find the audio segment whose relevance to the topic of interest is within a preset threshold range, and obtain the corresponding matching time period;

[0140] The second media code stream corresponding to the matching time period is sent preferentially.

[0141] In one implementation, before the second sending module performs the process of analyzing the audio corresponding to the time period of interest to obtain the topic of interest, the second sending module further includes:

[0142] Determine whether the number of unsent second media code streams is gradually increasing;

[0143] When the number of unsent second media code streams increases gradually, the process of analyzing the audio corresponding to the interested time period to obtain the interested topic is performed.

[0144] In one implementation, the second sending module performs the process of analyzing the audio in the second media code stream that has not yet been sent, searching for an audio segment whose relevance to the topic of interest is within a preset threshold range, and obtaining a corresponding matching time period, including:

[0145] Analyze the audio in each GOP in the second media code stream that has not yet been sent to obtain a topic analysis result of each GOP;

[0146] According to the topic analysis results of each GOP, searching for a GOP whose relevance to the topic of interest is greater than a preset threshold;

[0147] The time periods corresponding to all GOPs whose relevance to the topic of interest is greater than a preset threshold are determined as matching time periods.

[0148] In one implementation, the second sending module performs the process of preferentially sending the second media code stream corresponding to the matching time period, including:

[0149] In the second media code stream corresponding to the matching time period, the corresponding GOPs are sent in order according to the relevance of the topic analysis results of the respective GOPs to the topic of interest.

[0150] In one implementation, the second sending module performs the process of preferentially sending the second media code stream corresponding to the matching time period, including:

[0151] Receive the current playback time point of the receiving end;

[0152] Calculate, according to the current playback time point, a predicted playback time point at which the second media code stream corresponding to each matching time period is played at the receiving end;

[0153] Get network status;

[0154] Calculate, according to the network condition, a predicted receiving time point at which the second media code stream corresponding to each matching time period arrives at the receiving end;

[0155] In the second media code streams corresponding to the matching time period, the second media code streams whose predicted playback time point is later than the predicted reception time point in the second media code streams corresponding to the matching time period are preferentially sent.

[0156] In one implementation, the second sending module performs the process of analyzing the audio corresponding to the time period of interest to obtain the topic of interest, including:

[0157] When no topic of interest can be obtained by analyzing the audio corresponding to the time period of interest, extending the length of the time period of interest to obtain an extended time period of interest;

[0158] The audio corresponding to the extended time period of interest is analyzed to obtain topics of interest.

[0159] In one implementation, the second sending module, when executing the process of sending the cached second media code stream to the receiving end, further includes:

[0160] When there is no matching time period or there is no corresponding second media code stream in the matching time period, the cached second media code stream is sent to the receiving end in sequence.

[0161] Corresponding to the method of the second embodiment, the present invention further provides a multimedia data transmission control system 6, comprising: a transmitting end and a receiving end. The transmitting end comprises the transmitting end multimedia data transmission control device described above, and the receiving end comprises a receiving end multimedia data transmission control device, and the receiving end multimedia data transmission control device comprises:

[0162] A receiving module, used for receiving and storing the first media code stream and the second media code stream;

[0163] The processing module is used to determine whether the current continuous playback duration exceeds the threshold duration when receiving the playback instruction. If the current continuous playback duration exceeds the threshold duration, the interested time period is obtained according to the current playback time point and sent to the sending end.

[0164] In this device, while ensuring that the low-quality first media code stream can be sent smoothly, the cached high-quality second media code stream is selected and prioritized. The receiving end obtains the user's interested time period by observing the user's continuous playback duration. The sending end analyzes and compares the audio data of the user's interested time period captured by the receiving end with the audio of the second media code stream that has not been sent, finds the part with a high matching degree between the two in the second media code stream, and sends the content of this part first. This device allows users to watch high-quality and highly relevant playback videos during playback, improving the user's viewing experience.

[0165] Embodiment 4

[0166] An embodiment of the present invention further provides a storage medium on which computer instructions are stored. When the instructions are executed by a processor, the multimedia data transmission control method of any of the above embodiments is implemented.

[0167] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, random access memories (RAM), read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0168] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0169] Corresponding to the above-mentioned computer storage medium, in one embodiment, a computer device is also provided, which includes a memory, an encoder, and a computer program stored in the memory and executable on the encoder, wherein when the encoder executes the program, any one of the multimedia data transmission control methods in the above-mentioned embodiments is implemented.

[0170] The above computer device selects and prioritizes the cached high-quality second media streams while ensuring that the low-quality first media stream can be sent smoothly. The receiving end obtains the user's interested time period by observing the user's continuous playback duration. The sending end analyzes and compares the audio data of the user's interested time period captured by the receiving end with the audio of the second media stream that has not been sent, finds the part with a high degree of match between the two in the second media stream, and sends the content of this part first. The above computer device allows users to watch high-quality and highly relevant playback videos during playback, improving the user's viewing experience.

[0171] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multimedia data transmission control method, characterized in that: include: Receiving a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream; Sending the first media code stream to a receiving end; caching the second media code stream; When redundant bandwidth exists in the transmission network, sending the cached second media code stream to the receiving end; The process of sending the cached second media code stream to the receiving end includes: When receiving the time period of interest sent by the receiving end, analyzing the audio corresponding to the time period of interest to obtain the topic of interest; Analyze the audio in the second media stream that has not yet been sent, find the audio segment whose relevance to the topic of interest is within a preset threshold range, and obtain the corresponding matching time period; The second media code stream corresponding to the matching time period is sent preferentially.

2. The multimedia data transmission control method according to claim 1, characterized in that: Before the process of analyzing the audio corresponding to the time period of interest to obtain the topic of interest, the process further includes: Determine whether the number of unsent second media code streams is gradually increasing; When the number of unsent second media code streams increases gradually, the process of analyzing the audio corresponding to the interested time period to obtain the interested topic is performed.

3. The multimedia data transmission control method according to claim 1, characterized in that: The process of analyzing the audio in the second media code stream that has not yet been sent, searching for an audio segment whose relevance to the topic of interest is within a preset threshold range, and obtaining a corresponding matching time period includes: Analyze the audio in each GOP in the second media code stream that has not yet been sent to obtain a topic analysis result of each GOP; According to the topic analysis results of each GOP, searching for a GOP whose relevance to the topic of interest is greater than a preset threshold; The time periods corresponding to all GOPs whose relevance to the topic of interest is greater than a preset threshold are determined as matching time periods.

4. The multimedia data transmission control method according to claim 3, characterized in that: The process of preferentially sending the second media code stream corresponding to the matching time period includes: In the second media code stream corresponding to the matching time period, the corresponding GOPs are sent in order according to the relevance of the topic analysis results of the respective GOPs to the topic of interest.

5. The multimedia data transmission control method according to claim 1, characterized in that: The process of preferentially sending the second media code stream corresponding to the matching time period includes: Receive the current playback time point of the receiving end; Calculate, according to the current playback time point, a predicted playback time point at which the second media code stream corresponding to each matching time period is played at the receiving end; Get network status; Calculate, according to the network condition, a predicted receiving time point at which the second media code stream corresponding to each matching time period arrives at the receiving end; In the second media code streams corresponding to the matching time period, the second media code streams whose predicted playback time point is later than the predicted reception time point in the second media code streams corresponding to the matching time period are preferentially sent.

6. The multimedia data transmission control method according to claim 1, characterized in that: The process of analyzing the audio corresponding to the time period of interest to obtain the topic of interest includes: When no topic of interest can be obtained by analyzing the audio corresponding to the time period of interest, extending the length of the time period of interest to obtain an extended time period of interest; The audio corresponding to the extended time period of interest is analyzed to obtain topics of interest.

7. The multimedia data transmission control method according to any one of claims 1 to 6, characterized in that: The process of sending the cached second media code stream to the receiving end also includes: When there is no matching time period or there is no corresponding second media code stream in the matching time period, the cached second media code stream is sent to the receiving end in sequence.

8. A multimedia data transmission control method, characterized in that: include: A sending end receives a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream; The sending end sends the first media code stream to the receiving end; The sending end caches the second media code stream; When redundant bandwidth exists in the transmission network, the sending end sends the cached second media code stream to the receiving end; The receiving end receives and stores the received first media code stream and the second media code stream; When the receiving end receives the playback instruction, it determines whether the current continuous playback duration exceeds the threshold duration. If the current continuous playback duration exceeds the threshold duration, it obtains the time period of interest based on the current playback time point and sends it to the sending end; The process of sending the cached second media code stream from the sending end to the receiving end includes: When the sending end receives the time period of interest sent by the receiving end, the audio corresponding to the time period of interest is analyzed to obtain the topic of interest; The sending end analyzes the audio in the second media code stream that has not yet been sent, searches for an audio segment whose relevance to the topic of interest is within a preset threshold range, and obtains a corresponding matching time period; The sending end preferentially sends the second media code stream corresponding to the matching time period.

9. A sending end multimedia data transmission control device, characterized in that: include: A receiving module, configured to receive a first media code stream and a second media code stream encoded from the same original media file, wherein the quality of the second media code stream is higher than the quality of the first media code stream; A first sending module, used for sending the first media code stream to a receiving end; A cache module, used for caching the second media code stream; A second sending module, used for sending the cached second media code stream to the receiving end when there is redundant bandwidth in the transmission network; The operation of the second sending module in the process of sending the cached second media code stream to the receiving end includes: When receiving the time period of interest sent by the receiving end, analyzing the audio corresponding to the time period of interest to obtain the topic of interest; Analyze the audio in the second media stream that has not yet been sent, find the audio segment whose relevance to the topic of interest is within a preset threshold range, and obtain the corresponding matching time period; The second media code stream corresponding to the matching time period is sent preferentially.

10. A multimedia data transmission control system at the sending end, characterized in that: Includes: a sending end and a receiving end; Wherein, the transmitting end comprises the transmitting end multimedia data transmission control device as claimed in claim 9, and the receiving end comprises the receiving end multimedia data transmission control device; The receiving end multimedia data transmission control device comprises: A receiving module, used for receiving and storing the first media code stream and the second media code stream; The processing module is used to determine whether the current continuous playback duration exceeds the threshold duration when receiving the playback instruction. If the current continuous playback duration exceeds the threshold duration, the interested time period is obtained according to the current playback time point and sent to the sending end.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images