Media data transmission method, electronic device, storage medium and program product

Abstract

The invention provides a media data transmission method, electronic equipment, a storage medium and a program product. The media data transmission method comprises the following steps: acquiring media data to be transmitted; segmenting the to-be-transmitted media data to obtain a plurality of data segments corresponding to the to-be-transmitted media data; controlling to send a plurality of data fragments corresponding to the to-be-transmitted media data to a receiving terminal; if the retransmission indication is not triggered in the sending process of the plurality of data fragments, terminating the sending process after the plurality of data fragments are sent; and if the retransmission indication is triggered in the sending process of the plurality of data fragments, controlling to send the data fragments specified by the retransmission indication to the receiving terminal.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data transmission, etc., and particularly relates to a method for transmitting media data, an electronic device, a storage medium, and a program product. Background Art

[0002] With the development of audio-video communication technology, audio-video transmission is widely used in various fields, such as audio-video calls, remote meetings, online education, multimedia entertainment, etc. Audio-video transmission technology relies on a high-speed and low-latency data transmission network to ensure that audio-video data can be transmitted from the sending end to the receiving end in the shortest time.

[0003] In the prior art, audio-video data is usually encoded and packaged into data packets and then sent from the sending end to the receiving end through the network. If network fluctuations, congestion, or instability occur during the transmission process, packet loss may occur, and when packet loss occurs, the entire data packet usually needs to be retransmitted.

[0004] However, in large-scale audio-video data transmission, overall retransmission will cause delay accumulation, and thus the real-time performance of audio-video transmission is poor. Summary of the Invention

[0005] The present disclosure provides a method for transmitting media data, an electronic device, a storage medium, and a program product.

[0006] According to one aspect of the present disclosure, there is provided a method for transmitting media data, including: Obtaining media data to be transmitted; Segmenting the media data to be transmitted to obtain a plurality of data segments corresponding to the media data to be transmitted; Controlling to send the plurality of data segments corresponding to the media data to be transmitted to a receiving terminal; If the sending process of the plurality of data segments does not trigger a retransmission indication, terminating the sending process after the plurality of data segments are sent; If the sending process of the plurality of data segments triggers a retransmission indication, controlling to send the data segments specified by the retransmission indication to the receiving terminal.

[0007] According to the method for transmitting media data according to at least one embodiment of the present disclosure, after the controlling to send the plurality of data segments corresponding to the media data to be transmitted to the receiving terminal, it further includes: In response to the feedback media data sent by the receiving terminal according to the received data segments, controlling to play the feedback media data.

[0008] According to the method for transmitting media data according to at least one embodiment of the present disclosure, the controlling to play the feedback media data includes: Determine whether full-duplex media processing is supported; If supported, control the playback of the feedback media data.

[0009] According to the media data transmission method of at least one embodiment of the present disclosure, if not supported, the controlling the playback of the feedback media data further includes: Determine whether the acquisition of the media data to be transmitted is ongoing; If so, interrupt the acquisition of the media data to be transmitted; Control the playback of the feedback media data; In response to the completion of the playback of the feedback media data, continue to acquire the media data to be transmitted.

[0010] According to the media data transmission method of at least one embodiment of the present disclosure, the determining whether full-duplex media processing is supported includes: Obtain the field value of the field for indicating whether full-duplex media processing is supported from the feedback media data; Determine whether full-duplex media processing is supported according to the field value.

[0011] According to the media data transmission method of at least one embodiment of the present disclosure, the acquiring the media data to be transmitted includes: In response to the received acquisition instruction of the user input, determine whether there is an acquisition ability; If so, acquire the media data to be transmitted.

[0012] According to the media data transmission method of at least one embodiment of the present disclosure, before controlling the sending of multiple data segments corresponding to the media data to be transmitted to the receiving terminal, it further includes: marking the multiple data segments to obtain multiple marked segments; The controlling the sending of multiple data segments corresponding to the media data to be transmitted to the receiving terminal includes: controlling the sending of the multiple marked segments to the receiving terminal; Optionally, the marking the multiple data segments includes: Performing timestamp marking on the multiple data segments; or, Performing sequential marking on the multiple data segments; or, Performing eigenvalue marking on the multiple data segments; Optionally, the controlling the sending of multiple data segments corresponding to the media data to be transmitted to the receiving terminal includes: Adding the multiple data segments to a message queue; Taking out data segments from the message queue; Storing the taken-out data segments in a cache and controlling the sending to the receiving terminal; Optionally, controlling to send the data segment specified by the retransmission indication to the receiving terminal includes: Obtaining the data segment specified by the retransmission indication from the cache; Adding the data segment specified by the retransmission indication to the message queue; Taking out the data segment specified by the retransmission indication from the message queue; Controlling to send the data segment specified by the retransmission indication to the receiving terminal.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, including: a memory that stores execution instructions; and a processor that executes the execution instructions stored in the memory, such that the processor executes the media data transmission method of any one of the embodiments of the present disclosure.

[0014] According to still another aspect of the present disclosure, there is provided a readable storage medium in which execution instructions are stored, and when the execution instructions are executed by a processor, they are used to implement the media data transmission method of any one of the embodiments of the present disclosure.

[0015] According to yet another aspect of the present disclosure, there is provided a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the media data transmission method of any one of the embodiments of the present disclosure. Description of the Drawings

[0016] The drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, are used to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are included in this specification and form a part of this specification.

[0017] Figure 1 is a schematic diagram of an application scenario of the media data transmission method according to an embodiment of the present disclosure.

[0018] Figure 2 is the flow of the media data transmission method according to an embodiment of the present disclosure Figure 1 .

[0019] Figure 3 is the flow of the media data transmission method according to an embodiment of the present disclosure Figure 2 .

[0020] Figure 4 is Figure 3 the flow of the control playback method in the media data transmission method shown Figure 1 .

[0021] Figure 5 is Figure 3The flowchart of the control playback method in the media data transmission method shown Figure 2 。

[0022] Figure 6 is Figure 4 The flowchart of the duplex judgment method in the media playback method shown

[0023] Figure 7 is Figure 2 The flowchart of the acquisition method in the media data transmission method shown

[0024] Figure 8 is the flowchart of the media data transmission method of an embodiment of the present disclosure Figure 3 。

[0025] Figure 9 is Figure 1 The flowchart of the control sending method in the media data transmission method shown

[0026] Figure 10 is Figure 1 The flowchart of the control retransmission method in the media data transmission method shown

[0027] Figure 11 is the schematic flowchart of the media data transmission method according to the embodiment of the present disclosure

[0028] Figure 12 is the schematic block diagram of the structure of the media data transmission device of an embodiment of the present disclosure

[0029] Figure 13 is the schematic block diagram of the structure of the electronic device of an embodiment of the present disclosure Detailed implementation manners

[0030] The present disclosure will be further described in detail below with reference to the accompanying drawings and examples. It can be understood that the specific examples described herein are only used to explain the relevant content, rather than limiting the present disclosure. In addition, it should be noted that, for the sake of convenience of description, only the parts related to the present disclosure are shown in the accompanying drawings

[0031] It should be noted that, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The technical solutions of the present disclosure will be described in detail below with reference to the accompanying drawings and embodiments

[0032] Taking telemedicine consultation as an example, the high-definition video images and clear audio explanations at the expert side need to be transmitted to the receiving end at the hospital where the patient is located in real time and accurately for local doctors to refer to for diagnosis. When network fluctuations occur, for example, during peak hospital network hours, a large amount of medical data and other business data are transmitted in the network simultaneously, which may cause the network bandwidth to become instantly strained, leading to packet loss. When packet loss occurs, it is often necessary to retransmit the entire data packet, which not only occupies additional network bandwidth resources, further deteriorating the already strained network condition, but also causes a significant increase in data transmission latency, resulting in the audio and video received at the receiving end of the hospital where the patient is located to freeze, the picture to freeze, or even be interrupted for a long time, seriously affecting the communication efficiency between the expert and local doctors, and may further interfere with the accurate judgment and timely treatment of the patient's condition, greatly reducing the reliability and real-time nature of telemedicine.

[0033] To this end, the present disclosure proposes a media data transmission method, an electronic device, a storage medium, and a program product. For scenarios such as large-scale online meetings, video live broadcasts, or video on-demand, the present disclosure can be implemented through media data transmission software installed on a server. The media data transmission software can process concurrent access by multiple users through a distributed architecture, providing powerful computing capabilities and reliable bandwidth guarantees. For services with a wide distribution of users and the need to reduce transmission latency, such as live broadcasts and cloud games, the present disclosure can be implemented through media data transmission software installed on edge computing nodes. For scenarios such as personal live broadcasts, short video collection and upload, and real-time transmission of intelligent monitoring devices, the present disclosure can be implemented through media data transmission software installed on mobile terminal devices such as mobile phones, tablets, and cameras.

[0034] Figure 1 Fig. shows an application scenario of the media data transmission method according to an embodiment of the present disclosure. In this application scenario, it may include a sending end 100 and a receiving end 200, and the sending end 100 is connected to the receiving end 200 through a network. The sending end 100 is configured to process the media data to be transmitted and send it to the receiving end 200; the receiving end 200 is configured to receive, decode, and play (or give feedback).

[0035] For the convenience of description and to make the technical solutions of the specific embodiments of the present disclosure easier to understand, before describing the media data transmission method implemented by the present disclosure, the technical terms involved in the specific embodiments of the present disclosure are explained as follows: Media data refers to data stored, transmitted, or processed in digital or analog form, including audio, video, images, or text.

[0036] Full-duplex media processing refers to supporting media data to be processed simultaneously in two directions: sending (playing) and receiving (acquiring).

[0037] Figure 2The overall flowchart of the media data transmission method M100 according to an embodiment of the present disclosure is shown. As Figure 2 The shown media data transmission method includes steps S110 to S150. Among them, this method can be executed by electronic devices such as mobile phones, computers, servers, etc.

[0038] Specifically, Figure 2 The shown media data transmission method includes: Step S110, obtaining the media data to be transmitted.

[0039] In some embodiments of the present disclosure, the media data to be transmitted obtained through step S110 can be a type of media data or a combination of multiple types of media data; for example, the media data to be transmitted is "audio data" or "video data" or "audio data and video data".

[0040] The media data to be transmitted can be collected by hardware structures such as microphones, audio capture cards, cameras, etc. During the process of collecting the media data to be transmitted, the collection process can be controlled by audio or video control software, such as controlling start, pause, parameters (such as volume, gain, etc.).

[0041] Step S120, segmenting the media data to be transmitted to obtain multiple data segments corresponding to the media data to be transmitted.

[0042] In some embodiments of the present disclosure, step S120 can segment the media data to be transmitted based on a fixed time length, or based on predefined time points or marker points, or randomly or based on content, etc.

[0043] Among them, when segmenting the media data to be transmitted based on content, the media data to be transmitted can be first analyzed for content to obtain a content analysis result; then the media data to be transmitted is segmented according to the content analysis result. Taking the media data to be transmitted as audio data as an example, content analysis of the media data to be transmitted can include silence detection of the audio data (detecting whether there are continuous silent parts in the audio data, and the silent parts can be used as natural boundaries for segmentation), voice activity detection (detecting which parts of the audio data are speech and which parts are background noise, and segmenting based on this information), word boundary detection (detecting the boundaries of speech units in the audio data to avoid segmentation between these boundaries), or audio signal change detection (identifying the switching points of the audio data according to the spectral changes of the audio data, such as taking the positions of speaker changes or obvious changes in intonation as switching points and using the switching points as natural boundaries for segmentation).

[0044] Step S130, controlling to send multiple data segments corresponding to the media data to be transmitted to the receiving terminal.

[0045] In some embodiments of the present disclosure, for scenarios with relatively small data segments, step S130 may control the transmission of multiple data segments to the receiving terminal via HTTP or HTTPS; for scenarios with high real-time requirements, step S130 may control the transmission of multiple data segments to the receiving terminal via the WebSocket protocol; for scenarios with real-time streaming media and low-latency requirements (such as Voice over Internet Protocol (VoIP) voice calls, video conferences, etc.), step S130 may control the transmission of multiple data segments to the receiving terminal via the Real-Time Transport Protocol (RTP), etc.

[0046] For scenarios that require guaranteed ordered transmission, step S130 may control the transmission of only one data segment each time; for scenarios of large-scale data transmission, step S130 may control the packaging of several data segments into one data packet for transmission, or may also control the use of pipelined transmission.

[0047] Step S140, if the transmission process of multiple data segments does not trigger a retransmission indication, terminate the transmission process after the transmission of multiple data segments is completed.

[0048] In some embodiments of the present disclosure, if the transmission process of multiple data segments does not trigger a retransmission indication, it indicates that the transmission process of multiple data segments is smooth and the overall transmission quality is good. The transmission process can be terminated after the transmission of multiple data segments is completed to achieve media data transmission.

[0049] Step S150, if the transmission process of multiple data segments triggers a retransmission indication, control the transmission of the data segments specified by the retransmission indication to the receiving terminal.

[0050] In some embodiments of the present disclosure, during the process of transmitting data segments, some data segments may fail to reach the receiving terminal successfully due to network jitter, insufficient bandwidth, or routing congestion, etc., or may be damaged in content due to noise or hardware failures, etc. Therefore, there may be differences between the data segments controlled to be transmitted by step S130 and the data segments received by the receiving terminal.

[0051] When the receiving terminal determines that a certain data segment is lost or damaged based on the received data segments, the receiving terminal may generate a retransmission indication for the data segment and then send it to the media data transmission software, so that the media data transmission software can receive the retransmission indication sent by the receiving terminal. When the receiving terminal does not confirm the receipt of a certain data segment, the media data transmission software may trigger a retransmission indication at the timeout. That is, the retransmission indication can be triggered based on data segment loss, data segment error, or timeout.

[0052] This retransmission indication typically includes explicit data identifiers (such as sequence numbers, fragment indices, etc.) and status information (such as the cumulative number of lost packets, packet loss ratio, etc.), and is used to inform the media data transmission software which data fragments need to be resent.

[0053] In addition to the data identifier and status information, the retransmission indication may further include one or more of time information (such as the packet loss timestamp for facilitating debugging or logging by the media data transmission software, the retransmission time for ensuring that the media data transmission software understands the retransmission timing requirements of the receiving terminal, etc.), receive window information (used to inform the media data transmission software of the current buffer status or receiving capacity of the receiving terminal), check information (used to help the media data transmission software verify whether the lost data fragments are correctly transmitted), retransmission priority (used to assign priorities to different retransmitted data fragments), retransmission type or strategy (used to indicate which retransmission method the media data transmission software adopts), feedback acknowledgment (used for the receiving terminal to confirm the correctly received data fragments).

[0054] Among them, the receiving terminal can detect packet loss using sequence numbers. For example, an incrementing sequence number is introduced in the data fragment header, and the receiving terminal determines whether a packet is lost by comparing the sequence number of the current data fragment with that of the previous data fragment; the receiving terminal can also detect packet loss through timeout. For example, after the media data transmission software sends a data fragment, the receiving terminal needs to confirm the receipt of the data fragment within the timeout period. If the receipt is not received within the timeout, it is considered that a packet is lost; the receiving terminal can also detect packet loss by comparing the number of received data fragments with the expected number.

[0055] In some embodiments of the present disclosure, step S150 may first obtain the data fragments specified by the retransmission indication, and then control the sending of the data fragments specified by the retransmission indication to the receiving terminal. This sending process is the same as the process of sending multiple data fragments through step S130.

[0056] For the multiple data fragments corresponding to the media data to be transmitted, each data fragment of the media data transmission method provided by the present disclosure can be transmitted in parallel or independently, which makes the transmission process more efficient and improves the real-time performance of media data transmission, especially in the case of limited bandwidth resources; when retransmission is required, the lost data fragments can be accurately transmitted according to the retransmission indication, thereby reducing unnecessary retransmissions and further enhancing the real-time performance of media data transmission; this media data transmission method solves the problem in the prior art that in large-scale audio and video data transmission, overall retransmission will cause delay accumulation, and thus the real-time performance of audio and video transmission is poor.

[0057] In some embodiments of the present disclosure, the process of obtaining media data to be transmitted through step S110 can continue; when continuously obtaining media data to be transmitted through step S110, step S130 can be continuously used to control the sending of data segments, and when retransmission is required, step S150 can be used to control the implementation of retransmission. The process of obtaining media data to be transmitted through step S110 can be ended by a user instruction; that is, continuously obtain media data to be transmitted through step S110 until the user actively terminates the obtaining process and stops the above process.

[0058] In some embodiments of the present disclosure, after the retransmission instruction is triggered, the data segment specified by the retransmission instruction can be directly sent to the receiving terminal through the control of step S150, such as Figure 2 shown; after the retransmission instruction is triggered, step S150 can also first consider the congestion situation of the current network, then adjust the retransmission strategy according to the congestion situation, and control the sending of the data segment specified by the retransmission instruction to the receiving terminal based on the adjusted retransmission strategy; in order to avoid network congestion caused by frequent retransmissions, a retransmission count mechanism or a retransmission timeout mechanism can be set. After the retransmission instruction is triggered, step S150 can first determine whether the preset retransmission count or retransmission time has been reached. If not, it controls the sending of the data segment specified by the retransmission instruction to the receiving terminal. If it has been reached, retransmission is stopped. When retransmission is stopped, the receiving terminal can adopt technologies such as Forward Error Correction (FEC) and packet loss concealment to reduce the impact of data segment loss.

[0059] Among them, the preset retransmission count or retransmission time can be fixed or can be dynamically adjusted in real time according to real-time feedback such as the packet loss rate, delay, and bandwidth usage of the network; this dynamic adjustment can be achieved through some algorithms or protocols (such as the TCP slow start algorithm, etc.).

[0060] Increasing the retransmission count can improve the transmission reliability of data, especially when the network environment is unstable, to ensure the integrity of data; reducing the retransmission count can shorten the overall data transmission time and reduce system latency, which is suitable for application scenarios with high real-time requirements.

[0061] The retransmission time refers to the time waited after a packet loss. The retransmission time determines the delay of retransmission, which in turn affects the transmission quality, real-time performance, and utilization of network resources of the data. Adjusting the retransmission time is a technical choice that balances delay, bandwidth utilization, system load, and data reliability; increasing the retransmission time can improve the success rate and reduce network congestion, and reducing the retransmission time can improve real-time performance and recovery efficiency.

[0062] Furthermore, for the media data transmission method provided by the present disclosure, after step S150, it can further include, for example Figure 3Step S160 shown above.

[0063] Step S160, in response to the feedback media data sent by the receiving terminal according to the received data segments, controls the playback of the feedback media data.

[0064] In some embodiments of the present disclosure, in scenarios such as two-way interaction or real-time response of the media data transmission software application, the receiving terminal is used to control the return of media data (i.e., feedback media data) according to the received data segments; this feedback media data can be generated by the receiving terminal or obtained by the receiving terminal, etc. When generated by the receiving terminal, the receiving terminal is specifically used to determine whether recognizable content can be obtained after combining the received data segments; if recognizable content cannot be obtained, continue to receive data segments and repeat the above determination process; if recognizable content can be obtained, corresponding feedback media data can be generated based on the recognizable content. When obtained by the receiving terminal, the feedback media data can be collected by hardware structures of the receiving terminal such as microphones, audio capture cards, cameras, etc.

[0065] After the receiving terminal controls the return of the feedback media data, the media data transmission software can respond to the feedback media data sent by the receiving terminal according to the received data segments through step S160.

[0066] When the receiving terminal controls the return of the feedback media data, it can directly control the return of the complete feedback media data to the media data transmission software; in particular, to improve the transmission efficiency, the feedback media can also be segmented and then controlled for transmission, and the specific segmented transmission process is the same as the method provided in step S120.

[0067] In some embodiments of the present disclosure, the received feedback media data can be "audio" or "video" or "audio and video"; when the feedback media data includes audio, it can be an audio URL, data in Base64 format, etc.

[0068] In some embodiments of the present disclosure, the feedback media data can be played by audio and video playback hardware structures such as sound cards, speakers, graphics cards, etc.

[0069] Through step S160, two-way transmission of media data can be achieved, improving communication efficiency.

[0070] Regarding step S160, in some embodiments of the present disclosure, it may include steps S161 to S162 as Figure 4 shown above.

[0071] Step S161, determines whether full-duplex media processing is supported.

[0072] In some embodiments of the present disclosure, step S161 may determine whether full-duplex media processing is supported based on hardware support information and operating system information, such as confirming whether there are devices that can simultaneously process media input (acquisition) and output (playback), such as microphones and speakers (or headphones), and confirming whether the audio interface supports two-way audio streams; step S161 may also determine whether full-duplex media processing is supported based on application layer configuration and protocol support, such as checking whether full-duplex media processing is explicitly enabled in the media data transmission software and checking whether the protocol supports full-duplex media processing.

[0073] In some cases, even if other factors such as hardware, operating system, application layer configuration, protocol, etc. support full-duplex media processing, whether full-duplex media processing can be successfully enabled depends on the processing capacity of the receiving terminal. At this time, step S161 may determine whether full-duplex media processing is supported according to the processing capacity information of the receiving terminal.

[0074] If it is determined through step S161 that full-duplex media processing is supported, step S162 is executed.

[0075] Step S162, control the playback of feedback media data.

[0076] In some embodiments of the present disclosure, supporting full-duplex media processing means that the media data transmission software can control the playback of feedback media data while controlling the reception of media data to be transmitted. Therefore, if it is determined through step S161 that full-duplex media processing is supported, the playback of feedback media data can be directly controlled through step S162.

[0077] Through steps S161 to S162, the playback of feedback media data can be directly controlled in the case of supporting full-duplex media processing, improving the two-way processing efficiency of media data.

[0078] Regarding step S160, in some embodiments of the present disclosure, when it is determined through step S161 that full-duplex media processing is not supported, it may further include steps S163 to S166 as Figure 5 shown.

[0079] Step S163, determine whether the acquisition of media data to be transmitted continues.

[0080] In some embodiments of the present disclosure, not supporting full-duplex media processing means that the media data transmission software cannot control the playback of feedback media data while controlling the reception of media data to be transmitted, that is, only one action of controlling reception or controlling playback can be performed at the same time.

[0081] Step S163 can determine whether the action of obtaining the media data to be transmitted continues by checking the status of the acquisition unit; Step S163 can also determine whether the action of obtaining the media data to be transmitted continues by checking whether the media data to be transmitted continues to be newly added, etc.

[0082] When it is determined through Step S163 that the obtaining of the media data to be transmitted does not continue, the playback feedback media data can be directly controlled through Step S162; when it is determined through Step S163 that the obtaining of the media data to be transmitted continues, Step S164 is executed.

[0083] Step S164, interrupt the obtaining of the media data to be transmitted.

[0084] In some embodiments of the present disclosure, Step S164 can interrupt the obtaining of the media data to be transmitted through a status variable; Step S164 can also interrupt the obtaining of the media data to be transmitted through a signal or event mechanism, etc.

[0085] Among them, the status variable is a flag indicating whether it is currently necessary to obtain the media data to be transmitted; it can be of boolean type (true or false), integer type or enumeration value, depending on the complexity of the status to be managed; when the status variable is "interrupted", the obtaining logic is temporarily skipped, but the media data to be transmitted remains active.

[0086] A signal is a triggering mechanism in an operating system or application program used to notify a specific operation or status change; an event is a synchronization mechanism between threads or processes, often used in multi-threaded programming to coordinate behaviors using an event object; the specific implementation mechanism for interrupting the obtaining of the media data to be transmitted through a signal or event mechanism, etc. is: the main acquisition logic continuously monitors the signal or event status; when a specific signal or event is detected, the logic for interrupting or resuming the obtaining is executed.

[0087] Step S165, control the playback of the feedback media data.

[0088] In some embodiments of the present disclosure, the feedback media data in Step S165 can be played by an audio-video playback hardware structure such as a sound card, speaker, graphics card, etc.

[0089] Step S166, continue to obtain the media data to be transmitted in response to the completion of the playback of the feedback media data.

[0090] In some embodiments of the present disclosure, when Step S164 interrupts the obtaining through a status variable, Step S166 also continues to obtain the media data to be transmitted through the status variable; at this time, the status variable is set to "normal obtaining" to control the normal operation of the obtaining logic.

[0091] In the core logic for controlling acquisition and interruption based on state variables, it may include a process of determining whether to acquire the media data to be transmitted according to the state variable; if the state variable indicates permission to acquire, the acquisition logic is executed; if the state variable indicates suspension of acquisition, the acquisition logic is skipped or suspended.

[0092] When step S164 interrupts the acquisition through a signal or event mechanism, continuously monitor the signal or event status, and when it is feedback that the media data playback is completed, trigger step S166 to continue acquiring the media data to be transmitted.

[0093] Through steps S163 to S166, it is possible to achieve the control and playback of feedback media data in a media processing mode that does not support full duplex.

[0094] Regarding step S161, in some embodiments of the present disclosure, it may include steps S1611 to S1612 as Figure 6 shown.

[0095] Step S1611, obtain the field value of the field used to indicate whether full duplex media processing is supported from the feedback media data.

[0096] In some embodiments of the present disclosure, the processing capability of the receiving terminal can be represented by the field value of the field used to indicate whether full duplex media processing is supported. At this time, the received feedback media data includes this field value, so the field value can be directly obtained through step S1611.

[0097] Step S1612, determine whether full duplex media processing is supported according to the field value.

[0098] In some embodiments of the present disclosure, the field value can represent support for full duplex media processing by 1, true or full_duplex, and can represent non - support for full duplex media processing (only support for half - duplex media processing) by 0, false or half_duplex. Step S1612 can determine whether full duplex media processing is supported according to the specific numerical value of the field value.

[0099] Regarding step S110, in some embodiments of the present disclosure, it may include steps S111 to S112 as Figure 7 shown.

[0100] Step S111, in response to the received acquisition instruction input by the user, determine whether there is an acquisition capability.

[0101] In some embodiments of the present disclosure, the acquisition instruction input by the user is usually used to start the acquisition process of the media data to be transmitted; this acquisition instruction can be transmitted through a key, a text command, a UI button or other input methods.

[0102] In some embodiments of the present disclosure, step S111 may comprehensively determine whether there is an acquisition ability from multiple aspects such as hardware device information, software support information, resource status information, etc.

[0103] Among them, at the hardware device level, it is possible to check whether the audio-video input device exists, check whether the audio-video input device is occupied by other programs, check whether the audio-video input device supports the current sampling rate, number of channels, etc.; at the software support level, it is possible to check whether the driver of the audio-video input device is correctly installed, check whether the used audio-video library can normally initialize the audio-video input device, check whether there is access permission to access the audio-video input device, etc.; at the configuration and resource level, it is possible to attempt to initialize the audio-video input device to determine whether a successful status is returned, and check whether resources such as memory and CPU are sufficient to support real-time audio-video acquisition, etc. In particular, in addition to the above aspects, step S111 may further include dynamic status checks. For example, acquisition runs in a separate thread, and it is necessary to check whether the thread is normally started and running; confirm whether the audio-video input device can provide effective audio-video signals (such as not being completely silent or distorted, etc.).

[0104] If it is determined through step S111 that there is an acquisition ability, execute step S112; if it is determined through step S111 that there is no acquisition ability, the reason and solution to the problem can be clearly feedback to the user (such as clearly prompting the user why acquisition fails, whether the device is not connected, the permission is insufficient, or the device is occupied, etc.); if the audio-video input device is partially supported (for example, does not support the current sampling rate or multi-channels), the acquisition requirements can be reduced by reducing the sampling rate (such as reducing from 44.1 kHz to 16 kHz), reducing the number of channels (such as adjusting stereo to mono), adjusting the format (such as selecting other data formats supported by the audio-video input device), etc.; the detailed information of the acquisition ability check failure (such as check parameters, hardware device status, etc.) can also be recorded in a log file for subsequent analysis and repair; the user can also be allowed to choose to exit the media data transmission software or retry after the problem is solved.

[0105] Step S112, acquire the media data to be transmitted.

[0106] By performing the acquisition ability judgment before acquiring the media data to be transmitted through steps S111 to S112, errors or exceptions during the acquisition process can be avoided, and the reliability of the system can be improved.

[0107] Furthermore, the media data transmission method provided by the present disclosure may further include step S170 as Figure 8 shown before step S130.

[0108] Step S170, mark multiple data segments to obtain multiple marked segments.

[0109] In some embodiments of the present disclosure, step S170 may mark multiple data segments by means of timestamp marking, hash marking, random marking, sequential marking, eigenvalue marking, etc. For example: step S170 may specifically be to perform timestamp marking on multiple data segments, step S170 may also specifically be to perform sequential marking on multiple data segments, and step S170 may further specifically be to perform eigenvalue marking on multiple data segments, etc.

[0110] At this time, step S130 specifically is: controlling to send multiple marked segments to the receiving terminal; step S150 specifically is: if the sending process of multiple data segments triggers a retransmission indication, controlling to send the marked segments specified by the retransmission indication to the receiving terminal.

[0111] Taking incremental marking as an example, in response to the received marked segments, the receiving terminal can determine whether there are segments that need to be retransmitted according to whether the marks are continuous; taking timestamp marking as an example, in response to the received marked segments, the receiving terminal can determine whether retransmission is needed according to the interval of timestamps; taking eigenvalue marking, etc. as an example, the receiving terminal can combine other information such as sequence numbers, timestamps, redundant data, etc. to determine whether retransmission is needed.

[0112] In particular, during the process of controlling the sending of data segments, relevant control messages can also be periodically controlled to be sent to inform the receiving terminal of the segment marks that have been sent; in response to the received control messages, the receiving terminal updates the status of the received segments according to the control messages so as to timely detect situations such as packet loss.

[0113] Regarding step S130, in some embodiments of the present disclosure, it may include steps S131 to S133 as Figure 9 shown.

[0114] Step S131, adding multiple data segments to the message queue.

[0115] In some embodiments of the present disclosure, for simple applications, the message queue in step S131 may be a memory queue; for applications that require high availability, high throughput, or distributed systems, the message queue in step S131 may be a distributed message queue, such as RabbitMQ, Kafka, etc.

[0116] Step S132, taking out data segments from the message queue.

[0117] In some embodiments of the present disclosure, step S132 may take out data segments from the message queue by means of first in first out or priority, etc.

[0118] In a priority queue, retransmitted data segments usually need to be set a higher priority to ensure that they are processed earlier than normal data segments. For example, the message priority of retransmitted data segments can be set to high priority to ensure that their priority in the queue is higher than that of other normal data segments. Alternatively, the message priority of retransmitted data segments can be set to medium-high, so that the retransmitted data segments have priority over ordinary data segments, but not necessarily higher than all normal data segments.

[0119] Step S133: Store the retrieved data segment in the cache and control its transmission to the receiving terminal.

[0120] In some embodiments of the present disclosure, step S133 can sequentially control the transmission of the retrieved data segments to the receiving terminal one by one.

[0121] Steps S131 to S133 cache and control the transmission of the retrieved data segments, which can bring a smoother transmission process, higher reliability, and better network bandwidth management.

[0122] At this time, regarding step S150, in some embodiments of the present disclosure, it may include steps S151 to S154 as Figure 10 shown.

[0123] Step S151: Retrieve the data segment specified by the retransmission indication from the cache.

[0124] Step S152: Add the data segment specified by the retransmission indication to the message queue.

[0125] Step S153: Retrieve the data segment specified by the retransmission indication from the message queue.

[0126] In some embodiments of the present disclosure, when the message queue in step S152 is a first-in-first-out queue, the data segment specified by the retransmission indication can be directly added to the end of the message queue through step S152. After adding, the media data transmission software can first process the data segments before the data segment specified by the retransmission indication in the message queue in sequence until the data segment specified by the retransmission indication is processed through step S153.

[0127] Step S154: Control the transmission of the data segment specified by the retransmission indication to the receiving terminal.

[0128] Through steps S151 to S154, the data segments in the cache can be reasonably processed to ensure that the data segments can be retransmitted.

[0129] The media data transmission method provided by the present disclosure can ensure that data segments are quickly and reliably transmitted from the sender to the receiver. Moreover, through the data segment caching and recombination mechanism, it can reduce latency and packet loss, improving the real-time performance and stability of media data transmission. At the same time, the data directly connects to the receiver without passing through a third-party cloud service, which can reduce the data transfer nodes, simplify the data transmission link, and reduce the usage cost of the cloud service, as well as lower the overall data storage and transmission costs. When problems occur during the media data transmission process, it can also reduce the problem tracking cost and make fault troubleshooting more efficient.

[0130] This media data transmission method can be applied in fields such as real-time voice and video conferencing, streaming media transmission, real-time video monitoring, online games, live broadcasts, telemedicine, augmented reality, and virtual reality.

[0131] Figure 11 An exemplary flowchart implemented based on the media data transmission method of the present disclosure is shown.

[0132] Figure 11 In the shown flowchart, taking a real-time voice call system as an example, the media data transmission method may include: Step S210, obtaining audio data.

[0133] In some embodiments of the present disclosure, during the process of obtaining audio data, an analog-to-digital converter is usually used to convert analog audio signals into digital audio data.

[0134] Step S220, segmenting the audio data to obtain multiple data segments.

[0135] In some embodiments of the present disclosure, in order to improve the voice quality, before segmenting the audio data through step S220, the audio data can also be preprocessed first, such as noise cancellation, echo suppression, etc., and then the processed audio data is segmented through step S220.

[0136] Step S220 can segment the audio data according to the duration, such as each data segment containing 20 milliseconds of audio, etc.; step S220 can also segment the audio data according to the size, such as each data segment having a size of 100 kb, etc.

[0137] Step S230, marking the multiple data segments to obtain multiple marked segments.

[0138] In some embodiments of the present disclosure, step S230 can identify the order of the data segments through serial numbers and help the receiving terminal recombine the audio data in the correct order.

[0139] Step S240, sequentially adding the multiple marked segments to the message queue in order.

[0140] In some embodiments of the present disclosure, multiple marked segments are added to a message queue for queuing processing. The message queue helps manage the marked segments and provides buffering for subsequent transmission.

[0141] Step S250: Retrieve the marked segment from the message queue based on the first-in-first-out method.

[0142] Step S260: Store the retrieved marked segment in a cache and control its transmission to the receiving terminal.

[0143] Step S270: If the transmission process of the marked segment does not trigger a retransmission indication, terminate the transmission process after the transmission of multiple data segments is completed.

[0144] Step S280: If the transmission process of the marked segment triggers a retransmission indication, control the transmission of the data segments specified by the retransmission indication to the receiving terminal.

[0145] Through a series of technical means such as audio acquisition, segmentation, message queue, caching, and retransmission, the remote voice call system can ensure the efficient and real-time transmission and recovery of audio data; it can ensure that even in an unstable network environment, audio and video content can still be transmitted to the receiving terminal with high quality.

[0146] Based on any of the above embodiments, the present disclosure also provides a media data transmission device.

[0147] Figure 12 It is a structural schematic diagram of a media data transmission device according to an embodiment of the present disclosure.

[0148] As Figure 12 shown, the media data transmission device includes: A data acquisition module 110 for acquiring media data to be transmitted.

[0149] A data segmentation module 120 for segmenting the media data to be transmitted to obtain multiple data segments corresponding to the media data to be transmitted.

[0150] A data transmission module 130 for controlling the transmission of multiple data segments corresponding to the media data to be transmitted to the receiving terminal.

[0151] A transmission termination module 140 for terminating the transmission process after the transmission of multiple data segments is completed if the transmission process of the multiple data segments does not trigger a retransmission indication.

[0152] A data retransmission module 150 for controlling the transmission of the data segments specified by the retransmission indication to the receiving terminal if the transmission process of the multiple data segments triggers a retransmission indication.

[0153] The above-mentioned media data transmission device may be in the form of computer software, and each module of the above-mentioned media data transmission device may be implemented by computer software modules.

[0154] For the specific implementation process of the functions and roles of each module in the above-mentioned device, please refer to the implementation process of the corresponding steps in the above-mentioned method for details, which will not be elaborated here.

[0155] The execution subject of the media data transmission method in the specific implementation manner of the present disclosure may be an electronic device such as a mobile phone, a computer, or a server.

[0156] Therefore, based on any one of the above-mentioned embodiments, the present disclosure further provides an electronic device, which can execute the media data transmission method of any one of the above-mentioned embodiments described in the present disclosure.

[0157] Figure 13 It is a structural schematic diagram of an electronic device 1000 according to an embodiment of the present disclosure.

[0158] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture may include any number of interconnected buses and bridges, depending on the specific application of the hardware and the overall design constraints. The bus 1100 connects various circuits including one or more processors 1200, a memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0159] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one connection line is used in this figure, but it does not mean that there is only one bus or one type of bus.

[0160] The present disclosure also provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the above-mentioned method. The "readable storage medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples of the readable storage medium include the following: an electrical connection part with one or more wirings (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0161] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of the present disclosure are executed in whole or in part.

[0162] The computer program or instructions can be stored in a readable storage medium, or transmitted from one readable storage medium to another. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The readable storage medium can be any available medium that can be accessed, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it can also be an optical medium, such as a digital video disc; or it can be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0163] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0164] This disclosure is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a means for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or a means for implementing the functions specified in multiple blocks.

[0165] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction means that implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or a means for implementing the functions specified in multiple blocks.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or a means for implementing the functions specified in multiple blocks.

[0167] In the description of this specification, the description with reference to terms such as "one embodiment / way", "some embodiments / ways", "example", "specific example", or "some examples", etc., means that the specific features, structures, or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of this disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / way or example. Moreover, the specific features, structures, or characteristics described can be combined in a suitable manner in any one or more embodiments / ways or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments / ways or examples described in this specification and the features of different embodiments / ways or examples.

[0168] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0169] Those skilled in the art should understand that the above-described embodiments are merely for clearly illustrating the present disclosure and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications can be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A method for transmitting media data, characterized in that: include: Obtaining media data to be transmitted; Segmenting the media data to be transmitted to obtain a plurality of data segments corresponding to the media data to be transmitted; Controlling sending of a plurality of data segments corresponding to the media data to be transmitted to a receiving terminal; If the sending process of the multiple data segments does not trigger a retransmission indication, terminating the sending process after the sending of the multiple data segments is completed; as well as If the sending process of the multiple data segments triggers a retransmission indication, control the sending of the data segments specified by the retransmission indication to the receiving terminal.

2. The media data transmission method according to claim 1, characterized in that: After the control sends the plurality of data segments corresponding to the to-be-transmitted media data to the receiving terminal, the method further includes: In response to the received feedback media data sent by the receiving terminal according to the received data segment, playing of the feedback media data is controlled.

3. The media data transmission method according to claim 2, characterized in that: The controlling the playing of the feedback media data includes: determining whether full-duplex media processing is supported; and If supported, control the playing of the feedback media data.

4. The media data transmission method according to claim 3, characterized in that: If not supported, the controlling the playing of the feedback media data further includes: Determining whether the acquisition of the to-be-transmitted media data is ongoing; If yes, interrupt to obtain the media data to be transmitted; controlling the playing of the feedback media data; and In response to completion of playing of the feedback media data, continue to obtain the media data to be transmitted.

5. The media data transmission method according to claim 3, characterized in that: The determining whether full-duplex media processing is supported includes: Acquire, from the feedback media data, a field value of a field indicating whether full-duplex media processing is supported; and Whether full-duplex media processing is supported is determined based on the field value.

6. The method for transmitting media data according to any one of claims 1 to 5, characterized in that: The obtaining of the media data to be transmitted includes: In response to a received acquisition instruction input by a user, determining whether there is an acquisition capability; and If available, obtain the media data to be transmitted.

7. The media data transmission method according to any one of claims 1 to 5, characterized in that: Before the control sends the multiple data segments corresponding to the to-be-transmitted media data to the receiving terminal, the method further includes: marking the multiple data segments to obtain multiple marked segments; The controlling sending of the plurality of data segments corresponding to the to-be-transmitted media data to the receiving terminal comprises: controlling sending of the plurality of marker segments to the receiving terminal; Optionally, marking the multiple data fragments includes: Time stamping the multiple data fragments; or, Sequentially marking the multiple data fragments; or, Marking the multiple data segments with characteristic values; Optionally, the controlling sending a plurality of data segments corresponding to the to-be-transmitted media data to a receiving terminal includes: Adding the multiple data fragments to a message queue; Retrieving data segments from the message queue; and The retrieved data fragments are stored in a cache and controlled to be sent to a receiving terminal; Optionally, the controlling sending the data segment specified by the retransmission indication to the receiving terminal includes: Acquire the data segment specified by the retransmission indication from the cache; Adding the data segment specified by the retransmission indication to the message queue; Retrieving the data segment specified by the retransmission indication from the message queue; and Controlling sending of the data segment specified by the retransmission indication to the receiving terminal; Optionally, the retransmission indication is triggered based on data segment loss, data segment error or timeout.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the media data transmission method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the media data transmission method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the media data transmission method according to any one of claims 1 to 7 is implemented.

Images