Data processing method and device and electronic equipment

Abstract

The invention discloses a data processing method and device and electronic equipment. The method comprises the steps that real-time transport protocol (RTP) data packets are received, each RTP data packet comprises RTP header data and actual media data loads, and the RTP data packets comprise audio and video media data information; the RTP data packet is analyzed, voice load data and video load data are obtained, the coding format of the voice load data is a first coding format, and the first coding format is a coding format which does not support recognition by a voice recognition engine; after the voice load data is analyzed, the voice load data is converted from the first coding format to a second coding format, and the second coding format is a coding format which can be recognized by a voice recognition engine. According to the method and the device, the technical problem that audio and video streams in a 5G enhanced call network cannot be analyzed in real time in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular to a method, device and electronic equipment for data processing. Background Art

[0002] With the advancement of 5G network construction and large-scale deployment, 5G communication services are developing rapidly. 5G enhanced calls are an upgrade of basic call services by operators. Based on the audio and video channels of the IMS (IP Multimedia Subsystem) network, a series of call enhancement services are provided, such as call subtitles, fun calls and intelligent translation, to provide users with a new call experience with multimedia, visualization and full interaction. The 5G enhanced call network is based on the existing IMS audio and video call network architecture, and introduces two new capabilities, NCP (Network Control Point) entity and UMF (User Media Function) entity. Different from the existing network equipment, NCP can not only be used to control the interaction of telephone call signaling SIP (Session Initiation Protocol), but also complete the service control HTTP (HyperText Transfer Protocol) request distribution, while UMF is mainly responsible for the replication and synthesis of call audio and video media streams.

[0003] In order to realize 5G enhanced call services such as call subtitles, fun calls and intelligent translation, it is first necessary to perform real-time analysis of the audio and video media streams in 5G enhanced calls. In the 5G enhanced call network, the analysis of the media stream not only requires the separation of the media control description information and the payload data packet, but also the separate processing of the audio stream and the video stream according to the payload type, and then use different AI algorithms to integrate the AI ​​capabilities of the audio and video streams respectively, such as applying speech recognition to the analyzed audio stream and gesture recognition to the video stream. However, there is no real-time analysis method for audio and video streams under the 5G enhanced call network in the relevant technology, which completes the combination of audio and video media streams and AI capabilities and realizes the upgrade of the visual call experience of 5G enhanced calls.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, device and electronic device to at least solve the technical problem in the related art that it is impossible to perform real-time analysis of audio and video streams under the 5G enhanced call network.

[0006] According to one aspect of an embodiment of the present application, a method for data processing is provided, including: receiving a real-time transport protocol RTP data packet, wherein each RTP data packet contains RTP header data and actual media data payload, and the RTP data packet contains audio and video media data information; parsing the RTP data packet to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by a voice recognition engine; after parsing the voice payload data, converting the voice payload data from the first encoding format to a second encoding format, wherein the second encoding format is an encoding format that can be recognized by the voice recognition engine.

[0007] Optionally, before receiving the real-time transport protocol RTP data packet, the method also includes: receiving the session description protocol SDP description information sent by the terminal device when the 5G enhanced call starts; parsing the SDP description information to obtain the payload media type name, encoding name and audio clock frequency; determining the media information descriptor based on the payload media type name, encoding name and audio clock frequency; performing regular expression matching on the media information descriptor to obtain media information; and storing the audio clock frequency when the media information indicates that the audio encoding format of the voice payload data is the first encoding format.

[0008] Optionally, the voice load data is parsed, including: determining the data length of the voice load data; when the data length is greater than a first threshold, determining that the voice load data is non-silent data, performing a first processing on the non-silent data, and obtaining a first audio stream to be transcoded, wherein the encoding format of the first audio stream is a first encoding format; when the data length is less than the first threshold, determining that the voice load data is silent data, performing a second processing on the silent data, and obtaining a second audio stream to be transcoded, wherein the encoding format of the second audio stream is the first encoding format.

[0009] Optionally, a first processing is performed on the non-silent data, including: obtaining multiple voice frames in the non-silent data, wherein each voice frame includes a mode request field, a content information table and a voice data stream; processing the mode request field, the content information table and the voice data stream in each voice frame to obtain a first audio stream.

[0010] Optionally, the mode request field, content information table and voice data stream in each voice frame are processed to obtain a first audio stream, including: byte encoding the mode request field in each voice frame to obtain an encoding result string; when the value of the encoding result string is a second value, retaining the voice data stream; when the value of the mode request field is not the second value, re-encoding the voice data stream; parsing the content information table in each voice frame, wherein the content information table includes a tail frame identifier, a frame type and a valid identifier; determining a frame rate of the first encoding format based on the frame type, wherein the frame rate is used to verify the actual valid length of the voice frame; when the valid identifier indicates that the voice frame is valid and the voice frame is not the last frame, retaining the valid frame data based on the actual valid length, and reordering the voice frame, and determining the first audio stream based on the content information table and the valid frame data.

[0011] Optionally, the silence data is subjected to a second processing, including: adding header data and placeholders to the voice data stream in the silence data, and compensating for silence frame losses caused by differences in packet transmission frequencies.

[0012] Optionally, converting the voice payload data from a first encoding format to a second encoding format includes: obtaining the audio clock frequency and encoding information of the first encoding format; determining the transcoding parameters of the transcoding tool based on the audio clock frequency and the encoding information; determining the transcoding statement based on the transcoding parameters, and converting the voice payload data from the first encoding format to the second encoding format based on the transcoding statement.

[0013] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: a receiving module, used to receive a real-time transport protocol RTP data packet, wherein each RTP data packet contains RTP header data and actual media data load, and the RTP data packet contains audio and video media data information; a parsing module, used to parse the RTP data packet to obtain voice load data and video load data, wherein the encoding format of the voice load data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the voice recognition engine; a conversion module, used to convert the voice load data from the first encoding format to a second encoding format after parsing the voice load data, wherein the second encoding format is an encoding format that can be recognized by the voice recognition engine.

[0014] According to another aspect of the embodiment of the present application, an electronic device is also provided, including: a memory for storing program instructions; a processor, connected to the memory, for executing program instructions to implement the following functions: receiving real-time transport protocol RTP data packets, wherein each RTP data packet contains RTP header data and actual media data payload, and the RTP data packet contains audio and video media data information; parsing the RTP data packets to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the voice recognition engine; after parsing the voice payload data, converting the voice payload data from the first encoding format to a second encoding format, wherein the second encoding format is an encoding format that can be recognized by the voice recognition engine.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the above-mentioned data processing method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned data processing method when executed by a processor.

[0017] In an embodiment of the present application, by receiving a real-time transport protocol RTP data packet, wherein each RTP data packet contains RTP header data and actual media data payload, and the RTP data packet contains audio and video media data information; the RTP data packet is parsed to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the voice recognition engine for recognition; after parsing the voice payload data, the voice payload data is converted from the first encoding format to a second encoding format, wherein the second encoding format is an encoding format that is recognizable by the voice recognition engine, thereby achieving the purpose of enabling the voice data in the first encoding format to be successfully parsed by the voice recognition engine, thereby achieving the technical effect of providing a recognizable audio stream for smart call applications while ensuring the quality of 5G enhanced calls, thereby solving the technical problem in the related art that the audio and video streams under the 5G enhanced call network cannot be parsed in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1is a hardware structure block diagram of a computer terminal for implementing a data processing method according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a data processing method according to an embodiment of the present application;

[0021] Figure 3 It is a general architecture diagram of a method for real-time parsing and dynamic decoding of audio and video streams in a 5G enhanced call according to an embodiment of the present application;

[0022] Figure 4 It is a structural diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0026] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0027] IMS (IP Multimedia Subsystem): is a network architecture used to control, provide and manage multimedia communication services such as voice, video and data services, mainly used in new generation mobile communication networks such as 5G.

[0028] NCP (Network Control Point): In the embodiment of the present application, NCP is a newly introduced network element used to control the interaction of telephone call signaling SIP and the distribution of service control HTTP requests.

[0029] UMF (User Media Function): is another newly added functional network element in the embodiment of the present application, which is mainly responsible for copying and synthesizing the audio and video media streams in the call.

[0030] SIP (Session Initiation Protocol): is a signaling protocol used to initiate, manage, and terminate telephone calls and multimedia sessions (such as video conferencing).

[0031] HTTP (HyperText Transfer Protocol): is an application layer protocol used to transmit hypertext documents, such as HTML pages. In the embodiment of the present application, HTTP is used to distribute service control requests.

[0032] PCM (Pulse Code Modulation): It is a digital signal encoding method used to convert analog signals into digital signals. It is commonly used in speech recognition and audio processing.

[0033] ffmpeg: is an open source computer program that can be used to record and convert digital audio / video and convert it into streams. It contains various multimedia-related tools that can be used to process the decoding, encoding, transcoding, and multiplexing of audio and video files.

[0034] RTP (Real-time Transport Protocol): is a network protocol used to transmit real-time data such as audio, video or analog data. It is used in conjunction with RTCP (Real-time Transport Control Protocol) to provide quality feedback and monitoring for multimedia data transmission.

[0035] SDP (Session Description Protocol): is a protocol that describes the format and parameters of media streams in a session (such as audio and video conferencing). It is usually included in SIP signaling and is used to negotiate the encoding format, address information, and other media characteristics of audio and video streams.

[0036] In the related technology, in the 5G enhanced call network, it is also necessary to dynamically decode the audio stream in the 5G enhanced call. The most common audio stream encoding form in the 5G enhanced call network is the adaptive multi-rate broadband (Adaptive Multi-Rate Wideband, hereinafter referred to as AMR-WB) encoding technology, because of its high coding efficiency and good error resistance, it can dynamically adjust the bit rate under different network conditions. However, such mainstream AMR-WB encoding has brought troubles to the recognition and parsing of audio streams of intelligent voice services such as intelligent translation and voice emoticons in fun calls in 5G enhanced calls. The audio stream format required by common voice recognition engines is PCM (Pulse Code Modulation). For AMR-WB encoded audio streams, all real-time voice recognition engines in related technologies do not support recognition. The inability to recognize voice directly affects the subsequent intelligent call interaction and hinders the installation and popularization of voice intelligent applications in 5G enhanced calls.

[0037] At present, the algorithm for transcoding AMR-WB encoded audio streams into recognizable PCM encoded streams is mainly for offline voice file transcoding, which is converted by adding AMR encoding packages to ffmpeg. However, in real-time audio and video communications, the media streams transmitted in the network are not directly accessible AMR-WB audio files. Conventional offline audio encoding conversion cannot handle the needs of voice recognition in real-time audio and video communications for intelligent interaction. If other encoding technologies are used for audio and video transmission in 5G enhanced calls, the call quality of 5G enhanced calls will be sacrificed, resulting in excessive audio noise, audio information loss and other problems, affecting the user experience.

[0038] In order to solve the problems existing in the related art, the present application embodiment provides a data processing method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is explained below.

[0039] The data processing method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 1As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0040] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0044] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0045] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for data processing. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Figure 2 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0047] Step S202, receiving a real-time transport protocol RTP data packet, wherein each RTP data packet includes RTP header data and actual media data payload, and the RTP data packet includes audio and video media data information.

[0048] In the above step S202, during the 5G enhanced call, the audio and video data stream is transmitted in real time through the network and encapsulated using the RTP protocol. The media server sets up a monitoring service. When the 5G enhanced call starts, the actual user call media stream will be sent to the IP and port specified by the exchange center server in the form of RTP data packets through the UDP transmission protocol. The media server receives the RTP data packets from the exchange center server in real time. These data packets contain the audio and video data in the real-time call. The RTP data packet consists of two parts, the RTP header data and the actual media data payload. The RTP header data (consisting of 12 bytes) contains control information, such as version (2 bits, indicating the version information of the RTP protocol), padding bit (1 bit, used to indicate whether there are additional padding bytes), extension bit (1 bit, used to indicate whether there is an RTP header extension), CSRC count (4 bits, indicating the number of CSRC identifiers), tag bit (1 bit, application-defined tag), payload type (7 bits, used to indicate the payload type. After extracting this information, it is combined with SDP media information to assist in verifying the current data media encoding type), sequence number (16 bits, used to identify the order of data packets. This sequence number is used for verification during actual splicing to prevent the disorder of data packet sending and receiving order from causing audio stream dislocation), timestamp (32 bits, used to synchronize the playback speed of the receiving end. By comparing two RTP packets with a sequence number difference of 1, the timestamp difference represents the RTP packet sending and receiving interval), synchronization signal (32 bits, used to identify the sender of the RTP packet) and CSRC identifier (32 bits, used to identify other sources involved in synchronization), etc. This part of information provides the sending order, timestamp, synchronization signal, etc. of the data packet, helping the receiving end to correctly decode and play the audio and video streams. The actual media data load of RTP contains specific audio and video data. In the embodiment of the present application, the audio stream is encoded using AMR-WB, and the video stream is encoded using H264. The data in these encoding formats are encapsulated in the payload part of the RTP data packet, and as the call proceeds, the media data payload will be continuously updated and sent.

[0049] For the above timestamps, if the timestamp interval is 20ms, it means that the RTP packet is sent at a frequency of 20ms, and 50 RTP packets are sent per second. The subsequent audio and video stream playback speed can be calculated based on this. The calculation formula is:

[0050] Audio and video rate (kbit / s) = audio and video payload data length (bit) / timestamp interval (ms)

[0051] The process of receiving RTP packets is the basis for real-time parsing and dynamic decoding of audio and video streams in 5G enhanced calls. By monitoring and receiving these packets, the media server can obtain audio and video stream information in real time and provide raw data for further parsing. The RTP header information provides the metadata required for decoding and playing audio and video streams, while the actual media data payload is the carrier of audio and video data. The combination of the two ensures real-time transmission and high-fidelity decoding of audio and video streams.

[0052] Step S204, parsing the RTP data packet to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the speech recognition engine.

[0053] In the above step S204, after the media server receives the RTP data packet, it needs to parse it to separate the audio and video payload data. Specifically, by parsing the payload type information in the RTP header, it can be identified whether the current data packet is loaded with audio data or video data. In 5G calls, audio is usually encoded in AMR-WB, that is, the first encoding format mentioned above, and video is encoded in H264, for example.

[0054] Step S206, after parsing the voice payload data, convert the voice payload data from the first encoding format to a second encoding format, wherein the second encoding format is an encoding format recognizable by the voice recognition engine.

[0055] In the above step S206, in order to enable the voice data to be processed by the voice recognition engine, the system needs to perform transcoding to convert the AMR-WB encoded voice payload data into the second encoding format, namely the PCM encoding format. The conversion process involves technical details such as the retention of valid frame data, the rearrangement of frame data, and the precise compensation of silent frames, which ensures the integrity and accuracy of the transcoded PCM audio data.

[0056] In step S202 of the above data processing method, before receiving the real-time transport protocol RTP data packet, the method also includes: when the 5G enhanced call starts, receiving the session description protocol SDP description information sent by the terminal device; parsing the SDP description information to obtain the load media type name, encoding name and audio clock frequency; determining the media information descriptor based on the load media type name, encoding name and audio clock frequency; performing regular expression matching on the media information descriptor to obtain media information; and when the media information indicates that the audio encoding format of the voice payload data is the first encoding format, storing the audio clock frequency.

[0057] In an embodiment of the present application, SDP is a protocol for describing a multimedia session, which is sent by a terminal device at the start of a 5G enhanced call and includes key parameters such as the media type, encoding format, and clock frequency of audio and video streams. This information is used by both terminals and network devices to negotiate the details of audio and video transmission to ensure that both parties can correctly decode and process media data. The media server needs to parse the received SDP description information first. The SDP description information contains multiple lines of media description, and the media server extracts key information in each line of description by matching regular expressions, such as the load media type name (used to distinguish audio or video), the encoding name (AMR-WB for audio, H264 for video), and the audio clock frequency (used to synchronize the playback rate of audio data). Based on the parsed load media type name, encoding name, and audio clock frequency, the system determines the media information descriptors, which contain the detailed configuration of the media stream. Through these descriptors, the media server can identify the specific encoding format and playback parameters of the audio and video streams. In order to accurately extract media information from the SDP description information, the media server uses regular expressions for matching. Regular expression is a pattern used for text matching. Through it, the server can quickly locate and extract the encoding information and clock frequency related to the audio and video streams. If the SDP description information indicates that the audio encoding format of the voice payload data is the first encoding format (AMR-WB), the system will store the audio clock frequency information. The audio clock frequency is a key parameter for decoding the audio stream, especially in the process of converting AMR-WB encoded audio to PCM format. It is used to adjust the decoding rate to ensure that the decoded audio can keep pace with the original audio and avoid audio and video asynchrony or audio distortion. In addition, the terminal device and the media server need to negotiate the address and port number for sending and receiving media streams, and separate the audio and video streaming data according to the load type.

[0058] In step S206 of the above data processing method, the voice load data is parsed, including: determining the data length of the voice load data; when the data length is greater than a first threshold, determining that the voice load data is non-silent data, performing a first processing on the non-silent data, and obtaining a first audio stream to be transcoded, wherein the encoding format of the first audio stream is the first encoding format; when the data length is less than the first threshold, determining that the voice load data is silent data, performing a second processing on the silent data, and obtaining a second audio stream to be transcoded, wherein the encoding format of the second audio stream is the first encoding format.

[0059] In an embodiment of the present application, the data length of the AMR-WB encoded voice payload data is first obtained. When the data length is greater than a first threshold value (such as 10 bytes), it indicates that the currently received data contains valid voice information and is a non-silent frame, that is, the voice payload data is non-silent data. The non-silent frame is subjected to a first processing, for example, including steps such as retaining valid frame data and rearranging frame data, to ensure that the AMR-WB encoded voice data can be correctly and completely extracted and retained, thereby obtaining a first audio stream to be transcoded.

[0060] When both parties of a call are in silence, the RTP data packet sent may be very short. When the data length is less than the first threshold (such as 10), it is a silent frame, and the silent data in the voice payload data needs to be subjected to a second processing. The second processing, for example, includes silent frame compensation, thereby obtaining a second audio stream to be transcoded. The encoding formats of the above-mentioned first audio stream and the second audio stream are both the first encoding format, namely the above-mentioned AMR-WB encoding format.

[0061] In the above steps, the non-silent data is first processed, including: obtaining multiple voice frames in the non-silent data, wherein each voice frame includes a mode request field, a content information table and a voice data stream; processing the mode request field, the content information table and the voice data stream in each voice frame to obtain a first audio stream.

[0062] In the above steps, the mode request field, content information table and voice data stream in each voice frame are processed to obtain the first audio stream, including: byte encoding the mode request field in each voice frame to obtain an encoding result string; when the value of the encoding result string is a second value, retaining the voice data stream; when the value of the mode request field is not the second value, re-encoding the voice data stream; parsing the content information table in each voice frame, wherein the content information table includes a tail frame identifier, a frame type and a valid identifier; determining the frame rate of the first encoding format according to the frame type, wherein the frame rate is used to verify the actual valid length of the voice frame; when the valid identifier indicates that the voice frame is valid and the voice frame is not the last frame, retaining the valid frame data according to the actual valid length, and reordering the voice frame, and determining the first audio stream according to the content information table and the valid frame data.

[0063] In the embodiment of the present application, each segment of non-silent data is composed of multiple voice frames, and each voice frame is composed of three parts: a mode request field (Codec Mode Request, hereinafter referred to as CMR), a content information table (The table of contents, hereinafter referred to as ToC), and a voice data stream.

[0064] Extract the voice data stream for each voice frame. Specifically, in each voice frame, the mode request field (CMR) is located in the first four bits, which is used to indicate the audio coding mode. First, parse the CMR to obtain the coding result string. When the value of the coding result string is the second value (for example, CMR=15), it indicates that the coding mode of the voice data stream has not been specially processed and can be directly retained without further conversion; if the value of CMR is not equal to the second value, it indicates that the voice data stream may use a non-AMR-WB coding mode. At this time, the voice data stream needs to be re-encoded to ensure that the data format is unified and can adapt to the subsequent transcoding process. For example, PCM transcoding can be performed in other ways, or it can be directly recognized by the speech recognition engine.

[0065] Then the ToC is parsed, and the ToC records the end frame identifier, frame type, and valid identifier. The end frame identifier (F) is used to mark whether the voice frame is the end frame of the current voice payload data; the frame type (FT) reflects the encoding format and rate of the frame, and is used to determine the frame rate of the first encoding format (AMR-WB), so as to verify whether the actual effective length of the voice frame is consistent with the expectation; the valid identifier is used to determine whether the voice frame is valid (i.e., whether it contains meaningful voice data). For each non-silent voice frame, if the valid identifier indicates that the voice frame is valid and the voice frame is not the end frame (i.e., F=0), the frame rate of the first encoding format (AMR-WB) is determined according to the frame type recorded in the ToC. This rate information is combined with the audio clock frequency determined by the SDP description information to verify whether the actual effective length of the received voice frame is correct. If the valid identifier indicates that the voice frame is valid, that is, the voice frame contains meaningful voice data, then the valid frame data will be retained according to the calculated actual effective length. After retaining the valid frame data, the system will reorder the voice frames and merge the ToC information and the valid frame data to construct the first audio stream for transcoding. This audio stream will maintain the AMR-WB encoding format, but will be parsed through the content information table and the extraction of valid voice bytes, making it more compatible with the subsequent PCM transcoding requirements.

[0066] In an embodiment of the present application, the 5th to 10th bits of the voice frame are ToC, and the 1st bit is the end frame flag F. When the end frame flag is 1, it means that this frame is not the last frame in the voice load data, and there are other frames behind it, which means that the voice load data is composed of multiple segments of frame data, each of which contains CMR information, ToC information, and voice data stream (or valid voice bytes). It is necessary to repeat the parsing of CMR and ToC for each segment, and splice the extracted valid voice bytes until the end frame flag in the ToC information is 0. Usually, in a single-channel single-frame scenario, there is only one content information table in a voice load data, and the end frame flag in the content information table must be 0, and repeated splicing is not required. The 6th bit is the valid (frame) flag, which determines whether the frame is a good frame, 1 for a good frame, and 0 for a bad frame. It is determined whether the bad frame needs to be retained based on the actual scenario. Bits 2 to 5 are the frame type FT. When FT is different, the frame rate of the effective audio AMR-WB encoding is also different. After obtaining the FT value, map and query the audio rate, and compare it with the audio rate calculated based on the extracted timestamp information to assist in checking whether the voice payload length obtained in the above process is missing. If there is a missing, fill it with a placeholder to ensure the audio playback speed. The comparison table of some frame types is as follows:

[0067]

[0068] Through accurate analysis of CMR and ToC, it is possible to determine whether the voice frame is valid, whether AMR-WB encoding is used, and its specific encoding rate, so as to perform effective data retention and re-arrangement.

[0069] In the above steps, the second processing is performed on the silent data, including: adding header data and placeholders to the voice data stream in the silent data, and compensating for the silent frame loss caused by the difference in packet transmission frequency.

[0070] In the embodiment of the present application, silent data generally refers to a data packet sent when there is no actual voice activity in a call, and its length is much shorter than that of a non-silent data packet, generally not exceeding 10 bytes. In an actual 5G enhanced call, there is a significant difference in the sending frequency of silent data packets and non-silent data packets. A non-silent normal data packet is sent every 20ms, and a silent data packet is sent every 160ms. Therefore, when the length is the same, the number of silent data packets sent is only one-eighth of the number of non-silent data packets sent. If the silent data is directly spliced ​​according to the number of packets sent, the silent duration obtained after splicing will be only one-eighth of the actual silent segment duration. If no special processing is performed, the length of the silent period in the transcoded audio stream will not accurately reflect the actual silent period in the call, resulting in damage to the integrity of the audio stream. Therefore, when it is detected that the current data packet is a silent data packet, the mode request field and the content information table are stripped, and the remaining bytes are in the AMR-WB data format, that is, the AMR header data is added to the voice data stream in the silent data packet, and 0 is used as a placeholder to ensure that the silent data packet matches the non-silent data packet in format, which is convenient for subsequent processing. And the missing 7 silent data packets are compensated and then spliced ​​into the AMR-WB audio stream. Specifically, since the sending frequency of silent data packets is much lower than that of non-silent data packets, directly splicing the received silent data packets into the audio stream will cause the silent period of the transcoded audio stream to be severely shortened. In order to compensate for this difference in packet sending frequency, the system needs to add an additional 7 silent frames (with 0 as placeholder) to each silent data packet after detecting the silent data packet according to the silent frame sending mechanism of AMR-WB, so that the total sending frequency of the silent data packet is equal to that of the non-silent data packet (both are sent every 20ms). This compensation mechanism ensures that the length of the silent period in the transcoded audio stream can accurately reflect the actual silence time in the call, avoids the loss of silent frames, and maintains the integrity and continuity of the audio stream. Accurate compensation of silent frames not only solves the problem of silent frame loss during the transcoding process, but also ensures the accuracy and integrity of the audio stream, providing high-quality basic audio data for subsequent speech recognition and intelligent applications. Through such a processing process (i.e., the second processing), even if there is a long period of silence during the call, the system can generate an audio stream that reflects the actual silent period, thereby ensuring that the intelligent voice service in the 5G enhanced call can run smoothly and accurately, improving the user's call experience.

[0071] The first processing and the second processing are performed alternately according to whether it is a silent data packet, until the tail frame is obtained, and then the spliced ​​AMR-WB audio stream (that is, the first audio stream and the second audio stream containing the converted audio data stream) is decoded.

[0072] In step S206 of the above data processing method, the voice load data is converted from the first encoding format to the second encoding format, including: obtaining the audio clock frequency and the encoding information of the first encoding format; determining the transcoding parameters of the transcoding tool based on the audio clock frequency and the encoding information; determining the transcoding statement based on the transcoding parameters, and converting the voice load data from the first encoding format to the second encoding format based on the transcoding statement.

[0073] In an embodiment of the present application, before the transcoding process begins, the system needs to extract the audio clock frequency and the encoding information of the first encoding format (AMR-WB) from the session description protocol (SDP) description information. The audio clock frequency is an important parameter for synchronizing the playback rate of audio data, and the encoding information clarifies the original encoding format of the voice payload data. After clarifying the audio clock frequency and the encoding information of the first encoding format, the next step is to determine the transcoding parameters of the transcoding tool (such as FFmpeg). The transcoding parameters include but are not limited to sampling rate, bit depth, number of channels, etc., which will directly affect the efficiency of the transcoding process and the audio quality of the transcoded output. The audio clock frequency is combined with the encoding information of the first encoding format to determine the most suitable transcoding parameters to ensure that the transcoded audio stream not only meets the requirements of the speech recognition engine, but also maintains the original quality of the audio. After the transcoding parameters are determined, the system will generate a transcoding statement based on these parameters. The transcoding statement is usually a command line instruction for instructing the transcoding tool (such as FFmpeg) how to convert the voice payload data encoded by AMR-WB to the PCM encoding format. For example, a transcoding statement may include options that specify parameters such as input format (AMR-WB), output format (PCM), sampling rate, bit depth, and number of channels. When a transcoding statement is determined, the system will execute the statement to convert the voice payload data from the first encoding format (AMR-WB) to the second encoding format (PCM).

[0074] In some embodiments of the present application, the transcoding parameters are determined based on the acquired audio clock frequency and encoding information, and the ffmpeg transcoding statements are spliced, such as:

[0075] ffmpeg-loglevel debug-re-f amr-acodec amr wb -stream loop -1-i pipe:0

[0076] -f s16le-acodesc pcm s16le -ar 16000-ac 1-y pipe:1

[0077] This conversion process is critical to enabling intelligent voice services in 5G enhanced calls. AMR-WB coding is very suitable for transmitting voice data in 5G networks due to its high efficiency and error resistance, but most speech recognition engines do not support direct recognition of AMR-WB encoded audio. By converting AMR-WB encoded voice data to PCM encoding format, it not only solves the compatibility problem with the speech recognition engine, but also ensures the quality of audio data during the transcoding process, providing high-quality audio input for subsequent speech recognition and intelligent applications (such as intelligent translation, speech-to-text, etc.).

[0078] Figure 3 This is an overall architecture diagram of a method for real-time parsing and dynamic decoding of audio and video streams in a 5G enhanced call according to an embodiment of the present application. Figure 3 It shows the complete process from incoming calls from VoLTE terminals to real-time parsing and dynamic decoding of audio and video streams, and finally output of PCM encoded audio streams that can be parsed by the speech recognition engine. Figure 1 Detailed explanation of each key component and step in:

[0079] 1. VoLTE terminals: VoLTE terminals represent user devices participating in 5G enhanced calls, such as smartphones or professional communication devices. These terminals conduct audio and video communications, incoming and outgoing data calls through the 5G network.

[0080] 2. Network transmission of call data packets: 5G network is responsible for transmitting call data packets generated by VoLTE terminals. These data packets contain encoded audio and video streams, in which voice data is encoded using AMR-WB and video data is encoded using H264.

[0081] 3. Switching center server: The switching center server is the core component in the 5G network, responsible for monitoring and receiving audio and video data packets from VoLTE terminals. When a 5G enhanced call starts, the server receives RTP data packets through the UDP transport protocol, which contain audio and video media payloads.

[0082] 4. Media Server: The media server plays an important role in 5G enhanced calls and is responsible for parsing, processing and transcoding received media data packets. It receives RTP data packets forwarded from the exchange center server and starts the real-time parsing process of audio and video streams.

[0083] 5. Real-time parsing of audio and video streams: In this process, the media server first reads the SDP description information after the audio and video media negotiation, extracts the audio stream encoding type (AMR-WB), audio clock frequency, and video encoding information (H264). Then, the server separates the control information and audio and video payload data according to the format of the RTP data packet, and separates the audio and video stream data packets according to the payload type for real-time parsing.

[0084] 6. AMR-WB voice data parsing and decoding process: For AMR-WB encoded voice payload data, the media server needs to further process it. Voice payload data is part of the RTP data packet and contains voice information encoded by AMR-WB. These data are extracted by the media server during the real-time parsing of the audio and video stream for use in the subsequent AMR-WB voice data parsing and decoding process. This process includes valid frame data retention, frame data reordering, silent frame precise compensation, and AMR to PCM decoding. Through this series of operations, the integrity of the voice stream is ensured, and it is converted into a PCM encoding type that can be parsed by the speech recognition engine.

[0085] 7. PCM-encoded voice stream: PCM-encoded voice stream is the output of AMR-WB voice data after the dynamic decoding process of the media server. PCM (Pulse Code Modulation) is a lossless audio coding format that can be directly parsed by the speech recognition and interaction engine. The speech recognition and interaction engine is the core of intelligent applications in 5G enhanced calls, such as intelligent translation, speech-to-text and other functions. It receives the voice stream that has been converted to PCM encoding, performs speech recognition, and performs intelligent interaction based on the recognition results. This coding conversion ensures the accuracy and fluency of intelligent voice services.

[0086] Figure 4 is a structural diagram of a data processing device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0087] The receiving module 40 is used to receive real-time transport protocol RTP data packets, wherein each RTP data packet contains RTP header data and actual media data payload, and the RTP data packet contains audio and video media data information;

[0088] The parsing module 42 is used to parse the RTP data packet to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the speech recognition engine;

[0089] The conversion module 44 is used to convert the voice load data from a first encoding format to a second encoding format after parsing the voice load data, wherein the second encoding format is an encoding format recognizable by a voice recognition engine.

[0090] In the receiving module in the above-mentioned data processing device, the receiving module is also used to receive the session description protocol SDP description information sent by the terminal device when the 5G enhanced call starts; parse the SDP description information to obtain the load media type name, encoding name and audio clock frequency; determine the media information descriptor based on the load media type name, encoding name and audio clock frequency; perform regular expression matching on the media information descriptor to obtain media information; and store the audio clock frequency when the media information indicates that the audio encoding format of the voice load data is the first encoding format.

[0091] In the parsing module in the above-mentioned data processing device, the parsing module is also used to determine the data length of the voice load data; when the data length is greater than a first threshold, the voice load data is determined to be non-silent data, and a first processing is performed on the non-silent data to obtain a first audio stream to be transcoded, wherein the encoding format of the first audio stream is the first encoding format; when the data length is less than the first threshold, the voice load data is determined to be silent data, and a second processing is performed on the silent data to obtain a second audio stream to be transcoded, wherein the encoding format of the second audio stream is the first encoding format.

[0092] In the parsing module in the above-mentioned data processing device, the parsing module is also used to obtain multiple voice frames in non-silent data, wherein each voice frame includes a mode request field, a content information table and a voice data stream; the mode request field, the content information table and the voice data stream in each voice frame are processed to obtain a first audio stream.

[0093] In the parsing module in the above-mentioned data processing device, the parsing module is also used to byte-encode the mode request field in each voice frame to obtain an encoding result string; when the value of the encoding result string is a second value, retain the voice data stream; when the value of the mode request field is not the second value, re-encode the voice data stream; parse the content information table in each voice frame, wherein the content information table includes a tail frame identifier, a frame type, and a valid identifier; determine the frame rate of the first encoding format based on the frame type, wherein the frame rate is used to verify the actual valid length of the voice frame; when the valid identifier indicates that the voice frame is valid and the voice frame is not the last frame, retain the valid frame data based on the actual valid length, re-arrange the voice frame, and determine the first audio stream based on the content information table and the valid frame data.

[0094] In the parsing module in the above data processing device, the parsing module is also used to add header data and placeholders to the voice data stream in the silent data, and compensate for the silent frame loss caused by the difference in packet transmission frequency.

[0095] In the conversion module in the above-mentioned data processing device, the conversion module is also used to obtain the audio clock frequency and the encoding information of the first encoding format; determine the transcoding parameters of the transcoding tool based on the audio clock frequency and the encoding information; determine the transcoding statement based on the transcoding parameters, and convert the voice load data from the first encoding format to the second encoding format based on the transcoding statement.

[0096] It should be noted that Figure 4 The data processing device shown is used to perform Figure 2 The data processing method shown, therefore the relevant explanations in the above data processing method are also applicable to the data processing device, and will not be repeated here.

[0097] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory, and is used to execute program instructions to implement the following functions: receiving real-time transport protocol RTP data packets, wherein each RTP data packet contains RTP header data and actual media data payload, and the RTP data packet contains audio and video media data information; parsing the RTP data packets to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the voice recognition engine; after parsing the voice payload data, converting the voice payload data from the first encoding format to a second encoding format, wherein the second encoding format is an encoding format that can be recognized by the voice recognition engine.

[0098] It should be noted that the above electronic equipment is used to execute Figure 2 The data processing method shown, therefore the relevant explanations and instructions in the above data processing method are also applicable to the electronic device and will not be repeated here.

[0099] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following data processing method by running the computer program: receiving a real-time transport protocol RTP data packet, wherein each RTP data packet includes RTP header data and actual media data payload, and the RTP data packet includes audio and video media data information; parsing the RTP data packet to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the voice recognition engine; after parsing the voice payload data, converting the voice payload data from the first encoding format to a second encoding format, wherein the second encoding format is an encoding format that can be recognized by the voice recognition engine.

[0100] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The data processing method shown, therefore the relevant explanations in the above data processing method are also applicable to the non-volatile storage medium, and will not be repeated here.

[0101] The embodiments of the present application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data processing method in each embodiment of the present application.

[0102] The embodiments of the present application also provide a computer program, which, when executed by a processor, implements the steps of the data processing method in each embodiment of the present application.

[0103] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0104] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0109] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for data processing, characterized in that: include: Receive a real-time transport protocol (RTP) data packet, wherein each RTP data packet includes RTP header data and actual media data payload, and the RTP data packet includes audio and video media data information; Parsing the RTP data packet to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the speech recognition engine; After parsing the voice payload data, the voice payload data is converted from the first encoding format to a second encoding format, wherein the second encoding format is an encoding format recognizable by the voice recognition engine.

2. The method according to claim 1, characterized in that Before receiving the real-time transport protocol RTP data packet, the method further includes: When a 5G enhanced call starts, receiving the session description protocol SDP description information sent by the terminal device; Parsing the SDP description information to obtain the payload media type name, encoding name and audio clock frequency; Determine a media information descriptor according to the payload media type name, the encoding name and the audio clock frequency; Performing regular expression matching on the media information descriptor to obtain media information; When the media information indicates that the audio encoding format of the voice payload data is the first encoding format, the audio clock frequency is stored.

3. The method according to claim 2, characterized in that Parsing the voice load data includes: Determining the data length of the voice payload data; When the data length is greater than a first threshold, determining that the voice payload data is non-silent data, performing a first process on the non-silent data to obtain a first audio stream to be transcoded, wherein the encoding format of the first audio stream is the first encoding format; When the data length is less than the first threshold, it is determined that the voice load data is silent data, and a second process is performed on the silent data to obtain a second audio stream to be transcoded, wherein the encoding format of the second audio stream is the first encoding format.

4. The method according to claim 3, characterized in that The first processing is performed on the non-silent data, including: Acquire multiple voice frames in the non-silent data, wherein each voice frame includes a mode request field, a content information table, and a voice data stream; The mode request field, the content information table and the voice data stream in each voice frame are processed to obtain the first audio stream.

5. The method according to claim 4, characterized in that Processing the mode request field, the content information table, and the voice data stream in each voice frame to obtain the first audio stream includes: Perform byte encoding on the mode request field in each speech frame to obtain an encoding result character string; When the value of the encoding result character string is a second value, retaining the voice data stream; When the value of the mode request field is not the second value, re-encoding the voice data stream; Parsing a content information table in each voice frame, wherein the content information table includes a tail frame identifier, a frame type, and a valid identifier; Determining a frame rate of the first coding format according to the frame type, wherein the frame rate is used to verify an actual effective length of the speech frame; When the valid identifier indicates that the voice frame is valid and the voice frame is not the last frame, the valid frame data is retained according to the actual valid length, and the voice frame is rearranged, and the first audio stream is determined according to the content information table and the valid frame data.

6. The method according to claim 3, characterized in that The second processing is performed on the silence data, comprising: Header data and placeholders are added to the voice data stream in the mute data, and the mute frame loss caused by the difference in packet sending frequency is compensated.

7. The method according to claim 2, characterized in that: Converting the voice payload data from the first coding format to a second coding format includes: Acquire encoding information of the audio clock frequency and the first encoding format; Determining transcoding parameters of a transcoding tool according to the audio clock frequency and the encoding information; A transcoding statement is determined according to the transcoding parameter, and the voice payload data is converted from the first encoding format to the second encoding format according to the transcoding statement.

8. A data processing device, characterized in that: include: A receiving module, used for receiving a real-time transport protocol RTP data packet, wherein each RTP data packet includes RTP header data and actual media data payload, and the RTP data packet includes audio and video media data information; A parsing module, used for parsing the RTP data packet to obtain voice payload data and video payload data, wherein the encoding format of the voice payload data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the speech recognition engine; The conversion module is used to convert the voice load data from the first encoding format to a second encoding format after parsing the voice load data, wherein the second encoding format is an encoding format recognizable by the voice recognition engine.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is connected to the memory and is used to execute program instructions to implement the following functions: receiving a real-time transport protocol RTP data packet, wherein each RTP data packet contains RTP header data and actual media data load, and the RTP data packet contains audio and video media data information; parsing the RTP data packet to obtain voice load data and video load data, wherein the encoding format of the voice load data is a first encoding format, wherein the first encoding format is an encoding format that is not supported by the speech recognition engine; after parsing the voice load data, converting the voice load data from the first encoding format to a second encoding format, wherein the second encoding format is an encoding format that can be recognized by the speech recognition engine.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the data processing method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the data processing method described in any one of claims 1 to 7 is implemented.

Images