Streaming media processing method, transmission device and receiving device

Abstract

The present invention provides a method for processing streaming media, a transmitting device and a receiving device. The method for processing streaming media includes the following steps: encoding original media data into encoded streaming media. Adding media information to a media frame in the encoded streaming media to generate the added streaming media. The media information is used as the payload data of the media frame. The added streaming media is encapsulated into a streaming media data packet. In this way, the encoding and decoding and communication protocol are not affected and the compatibility requirements are met.

Description

Technical Field

[0001] The present invention relates to a digital media processing technology, and in particular, also relates to a streaming media processing method, a transmitting device and a receiving device. Background Art

[0002] When digital cameras, mobile phones or surveillance cameras generate and send standard audio or video streams, they need to provide additional media information to facilitate analysis or provide additional information when playing audio or video. These additional media information are not defined within the scope of the standard protocol. Although there are different additional practices on the market, most of them use limited transmission protocols or file formats, which may result in the inability to display media information or reduced compatibility. Summary of the invention

[0003] The present invention is directed to a streaming media processing method, a transmitting device and a receiving device, which can define accompanying data stream transmission media information outside the existing streaming media format specification.

[0004] According to an embodiment of the present invention, a method for processing streaming media includes (but is not limited to) the following steps: encoding original media data into encoded streaming media. Adding media information to the end of a media frame in the encoded streaming media to generate an added streaming media. The media information is used as payload data of the media frame. Encapsulating the added streaming media into a streaming media data packet.

[0005] According to an embodiment of the present invention, a method for processing streaming media includes (but is not limited to) the following steps: receiving a streaming media data packet via a network. Obtaining an attached streaming media from the streaming media data packet. Obtaining media information from the end of a media frame in the attached streaming media. The media information is used as payload data of a media frame in the attached streaming media, and the original data of the media frame is generated by encoding the original media data.

[0006] According to an embodiment of the present invention, the transmission device includes (but is not limited to) a memory, a digital signal processor, a transceiver and a main processor. The memory is used to store program code. The main processor is coupled to the memory, the digital signal processor and the transceiver, and is configured to load and execute the program code to perform the following steps: Encode the original media data into encoded streaming media through the digital signal processor. Add media information to the end of the media frame in the encoded streaming media through the transceiver to generate the attached streaming media. The media information serves as the load data of the media frame. Encapsulate the attached streaming media into a streaming media data packet.

[0007] According to an embodiment of the present invention, the receiving device transmission device includes (but is not limited to) a memory, a transceiver and a main processor. The memory is used to store program code. The main processor is coupled to the memory and the transceiver, and is configured to load and execute the program code to perform the following steps: receiving a streaming media data packet via a network through the transceiver. Obtaining the attached streaming media from the streaming media data packet through the transceiver. Obtaining media information from the end of the media frame in the attached streaming media. The media information is used as the payload data of the media frame in the attached streaming media, and the original data of the media frame is generated by encoding the original media data.

[0008] Based on the above, according to the streaming media processing method, the transmitting device and the receiving device of the embodiment of the present invention, the transmitting device uses the media information to be attached as the new data of the media frame, and the receiving device can extract the media information from the attached streaming media. In this way, it is not necessary to rely on the transmission protocol and the file format, and the additional information can be effectively brought to the receiving end or the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the present invention.

[0010] Figure 1 is a block diagram of components of a system according to an embodiment of the present invention;

[0011] Figure 2 is a flowchart of a method for processing streaming media according to an embodiment of the present invention;

[0012] Figure 3 A schematic diagram illustrating additional media information according to an embodiment of the present invention;

[0013] Figure 4 is the format of the media information according to the embodiment of the present invention;

[0014] Figure 5 The figure is a flow chart of a method for processing streaming media according to an embodiment of the present invention.

[0015] Description of Figure Numbers

[0016] 1: system;

[0017] 10: sending device;

[0018] 20: receiving device;

[0019] 11, 21: Digital signal processor;

[0020] 12, 22: main processor;

[0021] 13, 23: transceiver;

[0022] 14, 24: memory;

[0023] 15: Image capture unit;

[0024] 16: Radio;

[0025] 27: Display;

[0026] 28: Speaker;

[0027] 30: Network;

[0028] S210-S230, S510-S530: steps;

[0029] F: media frame;

[0030] A: Media information. DETAILED DESCRIPTION

[0031] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0032] Figure 1 is a block diagram of components of a system 1 according to an embodiment of the present invention. Figure 1 , the system 1 includes (but is not limited to) a sending device 10 , a receiving device 20 and a network 30 .

[0033] The sending device 10 may be a surveillance camera (also known as an IP camera), a server, or other computing device. The surveillance camera may be a security camera, a video doorbell, a dashboard camera, or a dashcam, or other cameras. The sending device 10 includes (but is not limited to) a digital signal processor (DSP) 11, a main processor 12, a transceiver 13, and a memory 14. The digital signal processor 11 is a processor dedicated to digital signal processing. For example, audio codec, video codec, or image codec. The main processor 12 is coupled to the digital signal processor 11. The main processor 12 may be a central processing unit (CPU), a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessors, programmable controllers, field programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), neural network accelerators, or other similar components or combinations of the above components. In one embodiment, the main processor 12 is used to execute all or part of the operations of the sending device 10, and can load and execute various program codes, software modules, files and data recorded in the memory 14. In one embodiment, the digital signal processor 11 is embedded in the main processor 12. In another embodiment, the functions of the digital signal processor 11 can be implemented by the main processor 12.

[0034] The transceiver 13 is coupled to the main processor 12. The transceiver 13 is, for example, a transceiver that supports a wired network such as Ethernet, an optical fiber network, or a cable, and may include (but not limited to) components such as a connection interface, a signal converter, and a communication protocol processing chip. The transceiver 13 may also be a transceiver that supports wireless networks such as Wi-Fi, the fourth generation (4G), the fifth generation (5G), or a later generation of mobile networks, and may include (but not limited to) components such as an antenna, a digital to analog / analog to digital converter, and a communication protocol processing chip. In one embodiment, the transceiver 13 is used to transmit or receive data via the network 30.

[0035] The memory 14 is coupled to the main processor 12. The memory 14 can be any type of random access memory (RAM), read only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD) or similar components. In one embodiment, the memory 14 is used to store program code, software modules, configurations, data (e.g., original media data, encoded streaming media, or media information, etc.) or files.

[0036] In one embodiment, the transmitting device 10 further includes an image capture unit 15. The image capture unit 15 is coupled to the main processor 12 via the digital signal processor 11. The image capture unit 15 may be an image sensor. In one embodiment, the image capture unit 15 is used to record a video to generate video data (i.e., original media data).

[0037] In one embodiment, the transmitting device 10 further includes a microphone 16. The microphone 16 may be a microphone of a dynamic type, a condenser type, or an electret condenser type. The microphone 16 may also be a combination of other electronic components, analog-to-digital converters, filters, and audio processors that can receive sound waves (e.g., human voices, ambient sounds, machine operation sounds, etc.) and convert them into audio signals. In one embodiment, the microphone 16 is used to receive the speaker's voice to generate audio data (i.e., original media data). The microphone 16 may have a built-in digital signal processor that processes or encodes sound signals.

[0038] The receiving device 20 may be a desktop computer, a laptop computer, a smart phone, a tablet computer, a server, a set-top box or other computing devices. The receiving device 20 includes (but is not limited to) a main processor 22, a transceiver 23 and a memory 24. In one embodiment, the receiving device 20 further includes a digital signal processor 21. The main processor 22 is coupled to the digital signal processor 21, the transceiver 23 and the memory 24. The implementation and functions of the digital signal processor 21, the main processor 22, the transceiver 23 and the memory 24 can refer to the description of the digital signal processor 11, the main processor 12, the transceiver 13 and the memory 14 respectively, and will not be repeated here.

[0039] In one embodiment, the receiving device 20 further includes a display 27. The display 27 is coupled to the digital signal processor 21. The display 27 can be a display using various display technologies such as a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), or a projector.

[0040] In one embodiment, the receiving device 20 further includes a speaker 28. The speaker 28 is coupled to the digital signal processor 21. The speaker 28 can be a speaker or a loudspeaker. In one embodiment, the speaker 28 is used to play sound. The display 27 and / or the speaker 28 can also be replaced by a video and / or audio output interface, such as a DVI or HDMI interface.

[0041] The network 30 may be a private network or a public network (eg, the Internet), or a combination of both.

[0042] Hereinafter, the method described in the embodiment of the present invention will be described with reference to various devices, components and modules in the system 1. Each process of the method can be adjusted according to the implementation situation, and is not limited thereto.

[0043] Figure 2 Flow chart of a method for processing streaming media according to an embodiment of the present invention. Figure 2 , the main processor 12 encodes the original (raw) media data into encoded streaming media (encoded streaming media) through the digital signal processor 11 (step S210). Specifically, the original media data can be video data (e.g., image RGB / YUV format) generated by the image capture unit 15 and / or audio data (e.g., sampled sound signal) generated by the receiver 16. The digital signal processor 11 can compress or encode the video data into a corresponding format based on video compression technologies such as H.264 / AVC, H.265 / HEVC, MJPEG, VP9, ​​AV1, or other video compression technologies. Alternatively, the digital signal processor 11 can compress or encode the audio data into a corresponding format based on audio compression technologies such as AAC, MP3, LDAC, or other audio compression technologies.

[0044] The main processor 12 adds media information (e.g., metadata) to the end of a media frame (e.g., an image frame or an audio sample) in the encoded streaming media to generate padded streaming media (step S220). Specifically, the encoded streaming media includes several media frames. The media frames may be video frames and / or audio frames. In one embodiment, the main processor 12 may detect a target object in the original media data or media frame. For example, the target object is a person, a package, an animal, or a vehicle. The main processor 12 may generate media information based on the detection result of the target object. The data type of the media information may be text, voice, and / or image. In one embodiment, the media information may include events, time, location, and / or direction related to the target object. For example, the time when a person enters the video, the position of the person, or the direction of the sound. In other embodiments, the media information may also be predefined or received from user input operations.

[0045] The main processor 12 uses the media information as the (new) data of the media frame. That is, the (new) data is added to the original data of the media frame to form a new media frame. The added streaming media includes the new media frame. In one embodiment, the main processor 12 may add the media information at the end of the media frame. That is, the media information is continued after the original data of the media frame. For example, Figure 3 This is a schematic diagram for explaining additional media information A according to an embodiment of the present invention. Figure 3 , media information A is added to the end of one or more media frames F. It is worth noting that each media information may correspond to the content of the attached media frame, and media information may be attached to a specific frame media according to actual needs.

[0046] In one embodiment, the header of the media information includes an information identifier. For example, "000" or "111". The header of the media information is located at the head end of the media information. The information identifier is used to distinguish the original data of the media frame from the media information. That is, starting from the head, when the information identifier is read from the media frame, the data behind is the media information.

[0047] Figure 4 is the format of the media information according to the embodiment of the present invention. Figure 4 , the media information includes an information identifier, an instruction, a padding length, data, a total length, and a terminator. However, the information format is not limited thereto.

[0048] In one embodiment, the main processor 12 may modify the header of the media frame, and the modified content may be related to the additional media information, such as increasing the frame length or modifying the identifier.

[0049] Please refer to Figure 2, the main processor 12 encapsulates the attached streaming media into streaming media packets through the transceiver 13 (step S230). Specifically, the streaming media packets can comply with the Real-time Transport Protocol (RTP), the Real-time Streaming Protocol (RTSP) or other streaming media related protocols. Since the media information is part of the payload data of the media frame, it does not affect the streaming media encapsulation. The main processor 12 can transmit the streaming media packets via the network 30 through the transceiver 13.

[0050] Figure 5 Flow chart of a method for processing streaming media according to an embodiment of the present invention. Figure 5 , the main processor 22 receives the streaming data packet via the network 30 through the transceiver 23 (step S510). The main processor 22 obtains the attached streaming media from the streaming data packet through the transceiver 23 (step S520). That is, the transceiver 23 receives and parses the streaming data packet using the same or compatible protocol. For example, the transceiver 23 parses the RTP packet and obtains the attached streaming media accordingly.

[0051] The main processor 22 obtains media information from the end of the streaming media to which the streaming media has been attached (step S530). As described above, the media information serves as the payload data of the media frame in the attached streaming media, and the original data of the media frame is generated by encoding the original media data. The main processor 22 extracts the media information from one or more media frames. In one embodiment, the header of the media information includes an information identifier. The main processor 22 can search for the information identifier in one or more media frames in the attached streaming media. The main processor 22 can distinguish the original data of the media frame from the media information based on the position of the information identifier. The media information is attached to the end of the media frame. As long as the information identifier is read from any media frame, the data after the start of the identifier in this information media frame is the media information, and the media information can be extracted accordingly.

[0052] In one embodiment, the main processor 22 can separate the media information from the attached streaming media to restore it to the encoded streaming media. That is, the media information in the attached streaming media is removed. If the transmitting device 10 modifies the header of the media frame when generating the attached streaming media, it can be restored together. For example, the frame length is modified to the length of the original data. This encoded streaming media can be used for playback. For example, the digital signal processor 21 decodes the encoded streaming media and plays the video data through the display 27 and / or plays the audio data through the speaker 28. For another example, the receiving device 20 acts as a streaming media server, and the encoded streaming media can be stored in the memory 24, and the main processor 22 transmits it to other devices through the transceiver 23 when needed.

[0053] In one embodiment, the main processor 22 provides media information according to the corresponding media frame during the process of playing the encoded streaming media on the display 27 and / or the speaker 28. That is, when playing to a specific media frame, the original additional media information is provided. For example, while the display 27 displays the video, event marking is performed in the video.

[0054] In one embodiment, the main processor 22 can directly play the attached streaming media through the display 27 and / or the speaker 28 or other playback devices. That is, the media information is not removed from the attached streaming media. Since the media information is attached to the end of the media frame, the media information does not affect the original data of the media frame, nor does it affect the decoding of the digital signal processor 21, so compatibility can be ensured.

[0055] In summary, in the streaming media processing method, transmission device and receiving device of the embodiment of the present invention, a method for transmitting media information along with a data stream is defined outside the existing streaming media format specification, and the media information is attached as the payload data of the media frame without affecting the encoding and decoding and data transmission method and communication protocol. In this way, the need for users and systems to synchronously understand the attached media information and streaming media data is solved.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing streaming media, characterized in that: include: Step 1: Encode the original media data into encoded streaming media via a compression format, wherein the encoded streaming media includes one or more media frames; Step 2: Detecting a target object in the original media data or the media frame, and generating media information according to a detection result of the target object; The media information includes events, time, location and / or direction related to the target object; The coded streaming media is formed into an appended streaming media by appending media information to the end of the original data of the payload data of the one or more media frames to generate one or more new media frames, wherein the media information is used as new data of the payload data of the one or more media frames, thereby forming the encoded streaming media into an appended streaming media; wherein the appended streaming media includes new media frames; Step 3: Treat the original data of the media information and the payload data of the one or more media frames as the complete payload data of each new media frame, and encapsulate the attached streaming media into a streaming media data packet. The header of the media information includes an information identifier, and the information identifier is used to distinguish the media information and the original data in the complete payload data of each new media frame according to the position of the information identifier.

2. The method for processing streaming media according to claim 1, characterized in that: Also includes: A header of the media frame is modified, wherein the modified content is related to appending the media information.

3. A method for processing streaming media, characterized in that: include: Step 1: receiving streaming media data packets via the network; Step 2: Obtaining the attached streaming media from the streaming media data packet; Step 3: obtaining media information from the end of the original data of the complete payload data of each of the one or more media frames in the attached streaming media, including: searching for an information identifier in the complete payload data of each of the one or more media frames in the appended streaming media, wherein the complete payload data includes the original data and the media information, and a header of the media information includes the information identifier; and Distinguishing the original data and the media information in the complete payload data of each of the one or more media frames according to the position of the information identifier, wherein the streaming media data packet is encapsulated from the attached streaming media according to the complete payload data of each of the one or more media frames of the attached streaming media, wherein the original data of the one or more media frames is generated by encoding the original media data, The media information includes events, time, location and / or direction related to the target object in the original media data or the media frame.

4. The method for processing streaming media according to claim 3, further comprising: Separating the media information from the complete payload data of each of the one or more media frames in the appended streaming media to restore the appended streaming media into an encoded streaming media; wherein the encoded streaming media is for playback; as well as During the process of playing the encoded streaming media, the corresponding separated media information is provided according to the corresponding media frame.

5. A conveying device comprising: A memory for storing program codes; Digital signal processor; Transceiver; as well as A main processor, coupled to the memory, the digital signal processor and the transceiver, is configured to load and execute the program code to perform the following steps: Step 1: Encoding the original media data into encoded streaming media via a compression format through the digital signal processor, wherein the original media data is encoded into one or more media frames of the encoded streaming media; Step 2: Detecting, by the main processor, a target object in the original media data or the media frame, and generating media information according to the detection result of the target object; The media information includes events, time, location and / or direction related to the target object; By the main processor, the encoded streaming media is formed into an appended streaming media by appending media information to the end of the original data of the payload data of the one or more media frames to generate one or more new media frames, wherein the media information is used as new data of the payload data of the one or more media frames, thereby forming the encoded streaming media into an appended streaming media; wherein the appended streaming media includes new media frames; and Step 3: by the transceiver, treating the original data of the media information and the payload data of the one or more media frames as the complete payload data of each new media frame, and encapsulating the attached streaming media into a streaming media data packet, The header of the media information includes an information identifier, and the information identifier is used to distinguish the media information and the original data in the complete payload data of each new media frame according to the position of the information identifier.

6. A receiving device, comprising: A memory for storing program codes; Transceiver; as well as A main processor, coupled to the memory and the transceiver, is configured to load and execute the program code to perform the following steps: Step 1: receiving a streaming media data packet via a network through the transceiver; Step 2: obtaining the attached streaming media from the streaming media data packet through the transceiver; as well as Step 3: Obtaining media information from the end of the original data of the complete payload data of each of one or more media frames in the attached streaming media by the main processor, including: searching for an information identifier in the complete payload data of each of the one or more media frames in the appended streaming media, wherein the complete payload data includes the original data and the media information, and a header of the media information includes the information identifier; and Distinguishing the original data and the media information in the complete payload data of each of the one or more media frames according to the position of the information identifier, wherein the streaming media data packet is encapsulated from the attached streaming media according to the complete payload data of each of the one or more media frames of the attached streaming media, wherein the original data of the one or more media frames is generated by encoding the original media data, The media information includes events, time, location and / or direction related to the target object in the original media data or the media frame.

7. The receiving device according to claim 6, wherein the main processor is further configured to execute: Separating the media information from the complete payload data of each of the one or more media frames in the appended streaming media to restore the appended streaming media into an encoded streaming media; wherein the encoded streaming media is for playback; and During the process of playing the encoded streaming media, the corresponding separated media information is provided according to the corresponding media frame.

Images