Dynamic code rate adjustment data transmission method based on HDMI (High Definition Multimedia Interface)

Abstract

A dynamic code rate adjustment data transmission method based on HDMI comprises the following steps: initializing a sending end MJPEG video encoder and a PCM audio encoder, and requesting to allocate an IP address; monitoring a multicast message sent by a receiving end, obtaining a paired receiving end and storing an IP address of the paired receiving end; starting an MJPEG video encoder and a PCM audio encoder, and setting a video encoding code rate, a frame rate, an audio encoding format and a bit rate; an HDMI input source signal is detected, if the HDMI input signal is detected, MJPEG and PCM data buffer memories are allocated, a video code rate code table is initialized and adjusted, and PCM coding and network sending processing and MJPEG coding and network sending processing are started; and circularly monitoring the multicast message of the receiving end, updating the IP address of the receiving end if the serial number in the received message is the same as the serial number in the sending end, and then continuously monitoring. Through congestion control in the data transmission process, the HDMI signal long-distance transmission quality and stability can be effectively improved, and the problem that long-distance transmission wiring is difficult is solved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a data transmission method for dynamic bit rate adjustment based on HDMI. Background Art

[0002] High Definition Multimedia Interface (HDMI) is a digital video and audio interface technology. It is a dedicated digital interface suitable for image transmission and can transmit both audio and image signals. Traditional HDMI-based data transmission has the following shortcomings: HDMI transmission distance is short, generally 10 meters. When the transmission distance exceeds 10 meters, signal amplifier hardware needs to be added, resulting in increased costs; HDMI transmitters and receivers cannot transmit data to each other. For example, the receiver can send keyboard and mouse data to the transmitter to control the transmitter's behavior; HDMI transmits raw data, such as 1080P, 60HZ, 24bit, at a rate of 373Mbit / s. This large amount of data can easily cause data congestion.

[0003] To solve the problem of HDMI transmission distance, the existing technologies generally have the following methods:

[0004] (1) Male-to-female extension cable: This is the simplest and most cost-effective method, connecting devices directly via a male-to-female HDMI extension cable. Ordinary copper HDMI extension cables are affordable, but their transmission distance is limited. High-quality cables can generally guarantee 15-20 meters of high-definition signal transmission, which is suitable for short-distance signal extension scenarios.

[0005] (2) HDMI signal amplifier: When the transmission distance of the HDMI cable exceeds the standard transmission distance (usually no more than 15 meters), a signal amplifier can be used to enhance the signal. It is placed in the middle of the HDMI cable to amplify and compensate the signal to ensure that the signal can be stably transmitted to the receiving device. The advantage of the signal amplifier is that it is relatively simple to install and does not require large-scale modification of the original line, but it does require an additional power supply.

[0006] (3) HDMI repeater: Similar in function to a signal amplifier, it is also used to enhance the HDMI signal and can be placed in the middle of an HDMI cable. It can receive and retransmit HDMI signals, thereby extending the signal transmission distance. Repeaters are widely used in scenarios where high signal quality is required, such as conference rooms and multimedia classrooms.

[0007] (4) HDMI fiber optic cable: Using optical fiber as the transmission medium, it is not subject to external electromagnetic interference, and the signal transmission attenuation is almost zero, making it easy to achieve ultra-long-distance transmission of hundreds of meters. Optical fiber HDMI cables have advantages such as light weight and bending resistance, and have great advantages in professional fields such as medical imaging and broadcasting. However, the price of optical fiber HDMI cables is relatively high, and the connection and wiring of optical fibers require certain skills and experience. Care should be taken to protect them during use and avoid large folds.

[0008] (5) HDMI over IP Extender: This device converts HDMI signals into TCP / IP protocol, which can be transmitted directly via a network cable or with the help of routers, switches, and other devices. This method can achieve multiple receivers per transmitter, supports long-distance transmission, and can utilize existing network infrastructure, offering high flexibility and scalability.

[0009] (6) HDMI Wireless Transmitter: Consists of a transmitter and a receiver, it can wirelessly transmit audio and video signals from a signal source (such as a computer, game console, set-top box, etc.) to a remote HDTV or projector. It can achieve lossless and delay-free transmission of 1080p / 60 and 3D, without the need for software installation, and is plug-and-play. Wireless transmission is free from the constraints of cables and has high flexibility and convenience, but it may be affected by signal interference, and transmission quality and stability may be reduced in some complex environments.

[0010] CN102931552B discloses an HDMI extension cable with signal amplification, which connects an HDMI repeater (HDMI amplifier) ​​between the video source and the display output device, and is equipped with a signal amplification circuit. Compared with ordinary HDMI cables, the cable is longer than ordinary HDMI cables, and the maximum cable length can reach 25 meters. However, it cannot meet the needs of some application scenarios that require longer distance transmission. The complex signal amplification circuit is set up inside, using a four-layer epoxy board and strict circuit design requirements, such as specific routing rules and matching resistors, resulting in high production costs and thus high product prices.

[0011] CN103986899A discloses an HDMI extender transmission system, in which the transmitting and receiving terminals are connected via a network cable. The receiving terminal RJ45 receives the network cable signal and outputs it to an external display device via HDMI. This invention has a maximum transmission distance of 150 meters. Because the signal is not re-encoded to reduce the data volume, the signal transmission effect is affected, especially over long distances. Secondly, the transmitting and receiving terminals contain numerous chips and complex circuit components and connections, which increases the difficulty of design and production, leading to higher product costs.

[0012] CN103219625B discloses a single-cable extender for HDMI and control signals. This invention discloses a single-cable extender for HDMI and control signals. Using lossless coding, the amount of data required to be transmitted is enormous, and signals are susceptible to interference over long distances, resulting in reduced signal transmission efficiency. Installation and wiring are difficult over long transmission distances, and the extended distance is still limited, unable to exceed 150 meters. Furthermore, the hardware circuit is complex and uses a relatively expensive FBPA, resulting in high costs.

[0013] The above-mentioned prior art methods for extending HDMI transmission distance have the following disadvantages:

[0014] (1) Male-to-female extension cable: When using a direct copper cable connection, the resistance of the copper cable will increase with the cable length, causing signal attenuation. High-frequency signals are more susceptible to electromagnetic interference and capacitance and inductance effects during long-distance transmission. Therefore, the transmission distance is limited. Generally, when transmitting a 1080p signal over a distance of more than 20 meters, high-quality HDMI copper cables may experience signal quality degradation, such as blurred images, color distortion, and flickering.

[0015] (2) Using fiber optic cables to extend HDMI transmission distances is expensive because of the complex manufacturing process and high technical content of the electrical conversion module. Fiber optic cables are fragile and easily damaged by external forces during installation and use.

[0016] (3) Add a signal amplifier between the output source and the input source to increase the HDMI transmission distance. While amplifying the signal, the signal amplifier also amplifies noise and interference signals. Its internal circuit design and component performance have certain limitations and cannot fully compensate for the signal attenuation caused by long-distance transmission. The amplifier requires an additional power supply, which is inconvenient in some wiring environments and increases the complexity and cost of device connection. As the transmission distance increases, the signal quality improvement effect gradually weakens. For transmissions exceeding a certain distance, the signal quality cannot be effectively improved, and problems may still occur with the picture and sound. In a complex wiring environment, power supply problems cause the amplifier to fail to work properly, affecting signal transmission.

[0017] (4) The improvement end converts the HDMI signal into a network signal for transmission, and the receiving end receives the network signal and then restores it to an HDMI signal and outputs it to the display device. This method depends on the performance of network equipment (such as routers, switches, etc.) and network cables. The network environment is complex and there are problems such as electromagnetic interference and signal congestion, which can easily affect the transmission quality. If there is no network congestion control mechanism when the data volume is large, it will cause the network to become more congested and the HDMI signal transmission delay will increase. Summary of the Invention

[0018] The present invention provides a data transmission method with dynamic bit rate adjustment based on HDMI. By controlling congestion during data transmission, the long-distance transmission quality and stability of HDMI signals can be effectively improved, and the problem of difficult long-distance transmission wiring can be solved.

[0019] A method for dynamic bit rate adjustment data transmission based on HDMI, comprising the following steps:

[0020] Step 1: Initialize the HDMI audio and video encoding hardware module and its driver at the sending end, which includes an MJPEG video encoder and a PCM audio encoder; the sending end requests an IP address, and proceeds to the next step after successfully obtaining the IP address;

[0021] Step 2: Monitor the multicast message sent by the receiver. By comparing the sequence number in the sender and the multicast message, obtain the paired receiver and store its IP address. If the paired receiver cannot be obtained, continue to monitor the multicast message.

[0022] Step 3. Start the MJPEG video encoder and PCM audio encoder, set the video encoding bit rate and frame rate, as well as the audio encoding format and bit rate;

[0023] Step 4: Detect the HDMI input source signal. If an HDMI input signal is detected, proceed to the next step. Otherwise, wait for the HDMI input source to be connected to the HDMI transmitter.

[0024] Step 5: Allocate MJPEG and PCM data buffer memory, initialize the code table for adjusting the video bit rate, and then start PCM encoding and network transmission processing, and MJPEG encoding and network transmission processing respectively;

[0025] Step 6: Circularly monitor the multicast messages sent by the receiving end. If the sequence number in the received message is the same as the sender's own, update the receiving end IP address and continue to monitor the multicast messages.

[0026] Furthermore, the PCM encoding and network transmission processing in step 5 includes:

[0027] (1) Establish a TCP network connection with the receiving end and determine whether the connection is successful. If it fails, close the network connection and try again until the TCP network connection is successfully established.

[0028] (2) Read the PCM data, segment it into segments according to the predetermined number of bytes, and add a CRC checksum to each data segment;

[0029] (3) Send the segmented PCM data to the receiving end through the established TCP connection;

[0030] (4) Determine whether the data is sent successfully. If the sending fails and the TCP connection is not disconnected, discard the current PCM data and re-collect the next frame of data; if the sending fails and the TCP connection is disconnected, return to step (1) to re-establish the TCP network connection; if the sending is successful, proceed to the next step;

[0031] (5) Check whether the receiving end IP address has been updated. If the receiving end IP address has been updated, close the current network connection and return to step (1) to re-establish a TCP connection with the receiving end. If the transmission is successful, return to step (2) and continue the subsequent PCM data collection, encoding and segmentation.

[0032] Furthermore, the predetermined number of bytes is 1400 bytes.

[0033] Furthermore, the MJPEG encoding and network transmission processing in step 5 includes:

[0034] (1) Establish a TCP network connection with the receiving end and determine whether the connection is successful. If it fails, close the network connection and try again until the TCP network connection is successfully established.

[0035] (2) Read the MJEPG data, segment it into segments according to the predetermined number of bytes, and add a CRC checksum to each data segment;

[0036] (3) Sending the segmented MJPEG data to the receiving end through the established TCP connection;

[0037] (4) Determine whether the data is sent successfully. If the sending fails, proceed to the next step; if the sending succeeds, proceed to step (8);

[0038] (5) Check the receiving end network. If it is disconnected from the receiving end network, proceed to the next step; otherwise, proceed to step (7).

[0039] (6) Close the network, close the TCP network connection with the receiving end, and go to step (1);

[0040] (7) Discard MJPEG encoded data, adjust the MJPEG encoding code table, and reduce the MJPEG bit rate;

[0041] (8) Check whether the receiving end IP address is updated. If the receiving end IP address has been updated, go to step (6), otherwise go to the next step;

[0042] (9) Adjust the video MJPEG encoding bit rate: Calculate the average data size A per frame based on the video encoding resolution, frame rate, and the preset maximum data volume per second. When X>A+λ*A, it indicates that the data volume is overloaded, and the code table should be adjusted to reduce the data volume per frame to reduce the network burden. When X<A-λ*A, it indicates that the data volume is too low, and the code table should be adjusted to increase the data volume per frame to improve the video quality. Where X is the current video frame size read, and λ is the adjustment coefficient.

[0043] Furthermore, the predetermined number of bytes is 1400 bytes, and the adjustment coefficient λ is 0.2.

[0044] Furthermore, the video MJPEG encoding bitrate is adjusted based on proportional, integral, and differential PID control. The goal of PID control is to stabilize the current frame data volume X near the target average value A. The adjustment amount ΔR is calculated by the following formula:

[0045]

[0046] Among them, the current error e(t) = XA, X is the current frame data volume, and A is the target average value;

[0047] Proportional coefficient K p Indicates the impact of the current error, the larger the error, the more radical the adjustment;

[0048] Integration coefficient K i Used to eliminate historical accumulated errors;

[0049] Differential coefficient K d Used to predict error change trends and suppress oscillations;

[0050] When the calculated ΔR>0 and is greater than a certain threshold value, the bit rate is increased;

[0051] When the calculated ΔR is less than 0 and its absolute value is greater than a certain threshold, the bit rate is reduced.

[0052] Furthermore, K p The value is 0.1, K i The value is 0.01, K d The value is 0.05.

[0053] Furthermore, the receiving end processing steps are also included:

[0054] (1) Power on and start the program initialization;

[0055] (2) Check whether the hardware configuration is a direct network connection scheme: Determine whether a direct network connection scheme is adopted by reading the hardware's GPIO L0 pin. When it is high, the receiving end and the sending end are directly connected, and the receiving end's IP address is determined. In a non-direct connection scheme, the router provides an IP address, and the IP address is obtained by polling until a valid IP address is obtained and then proceed to the next step;

[0056] (3) Initialize the multicast service, initialize the user interface UI, prompt the user that the application initialization is complete, and start receiving and playing MJPEG / PCM data;

[0057] (4) Check whether the sending end and the receiving end have established a connection. If so, proceed to step (6); otherwise, proceed to step (5);

[0058] (5) Display "No Signal" on the UI to prompt the user, indicating that the receiving end and the sending end have not yet established a connection, and then proceed to step (7);

[0059] (6) Hide the UI and display the received video;

[0060] (7) The current thread sleeps to reduce CPU usage;

[0061] (8) Check whether the time from the last broadcast of the device information has exceeded the predetermined time. If it has exceeded, go to step (9); otherwise, return to step (4);

[0062] (9) Send a multicast message through the multicast service created in step (3). When the sender receives a multicast message that matches its own sequence number, it obtains the IP address of the receiver from the multicast message. The sender and receiver establish a TCP connection and send data to each other. After sending the multicast message, return to step (3).

[0063] Furthermore, the method also includes the steps of receiving and playing MJPEG data and PCM data at the receiving end:

[0064] (1) Start the processing thread for receiving and playing MJPEG and PCM data;

[0065] (2) Initialize the TCP server and start the playback device of MJPEG and PCM data;

[0066] (3) The system enters a waiting state, waiting for the sender to request a TCP network connection;

[0067] (4) The TCP network connection between the sender and the receiver is successful. If the sender requests a connection and the connection is successful, proceed to step (5); otherwise, proceed to step (3).

[0068] (5) The receiving end receives the MJPEG and PCM segmented data sent by the sending end;

[0069] (6) Check whether the data is received successfully and determine whether the network protocol stack returns successfully. If successful, execute step (8); otherwise, execute step (7);

[0070] (7) Check whether the sending client is closed, check the returned error code, if it is "network is closed", execute step (3), otherwise execute step (5);

[0071] (8) Check whether the CRC of the segmented data packet header is correct and whether the segmented data is continuous. If the CRC is correct and the received data segment is continuous, execute step (10); otherwise, execute step (9);

[0072] (9) Discard the received audio and video data, release the occupied memory, and execute step (5).

[0073] (10) The received segmented data includes audio data and video data, and the segmented data is stored in the audio cache queue and the video cache queue;

[0074] (11) Check whether the sequence numbers of the data segments in the video cache queue and the audio cache queue are continuous, and whether the total number of received segments is equal to the total number of segments of the current frame stored in the segment header. If so, it indicates that a complete frame of data has been received, and step (12) is executed; otherwise, step (5) is executed;

[0075] (12) Playing MJPEG and PCM data: writing a frame of audio data or video data into the audio playback driver or video playback driver, and converting the audio and video data into HDMI signals, outputting them to the display device, and then executing step (5).

[0076] Technical effects of the present invention:

[0077] At a longer distance (such as in different rooms, floors or buildings), the sending end and the receiving end are connected through the existing network infrastructure (such as routers, switches). As long as they are in the same local area network, the devices at both ends can automatically discover and establish a TCP connection to realize the transmission of audio and video data and control information. In the face of a drop in transmission rate caused by external interference, the system reduces the bit rate through dynamic bit rate control, reduces the amount of data, and ensures low latency and video smoothness. This design allows long-distance high-quality transmission and simplifies wiring requirements, while maintaining stability and anti-interference capabilities during close direct connections. The data transmission method between the sending end and the receiving end in the present invention is not necessarily TCP, but can also be UDP, or other data communication protocols. The way the sending end and the receiving end access the infrastructure network is not limited to wired Ethernet, it can be WIFI. In addition, it has the following advantages:

[0078] 1. Dynamic Bitrate Control: When the network is idle, the system increases the bitrate to improve video quality. When the network is busy or external interference is detected, causing the transmission rate to drop, the dynamic bitrate control system reduces the bitrate, reducing the data volume and ensuring low latency and smooth video. This mechanism effectively alleviates network congestion and optimizes signal quality and stability over long-distance transmission.

[0079] 2. Ethernet-based HDMI extension solution: Unlike traditional physical extension methods (such as copper cables and fiber optic cables), this solution leverages existing Ethernet infrastructure to achieve long-distance transmission of HDMI signals. This approach not only reduces wiring complexity and costs, but also increases system flexibility, supporting automatic device discovery and connection establishment within the same LAN.

[0080] 3. Flexible Network Connection: This invention supports both direct network connection (i.e., direct connection between the sender and receiver via a standard network cable) and interconnection via existing network infrastructure such as routers and switches. The former is suitable for local network environments, while the latter is suitable for large-scale, multi-device scenarios, providing a wider range of application possibilities.

[0081] 4. Efficient Data Processing and Transmission Protocol: Based on PCM audio encoding and MJPEG video encoding, a reasonable segmentation strategy is designed for each frame of data, and a CRC checksum is added to ensure data integrity. In addition, special consideration is given to the success or failure of TCP connection establishment and the reconnection logic when the receiving end's IP address is updated, ensuring stable and reliable data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a schematic diagram of the structure of the transmitting end and the receiving end of the present invention;

[0083] Figure 2 This is a schematic diagram of a direct network cable connection between a transmitting end and a receiving end of the present invention;

[0084] Figure 3 It is a schematic diagram of the interconnection between the transmitting end and the receiving end of the present invention through the network infrastructure;

[0085] Figure 4 This is a flow chart of data processing at the sending end in the present invention;

[0086] Figure 5 This is a schematic diagram of the PCM encoding and network transmission processing process at the transmitting end in the present invention;

[0087] Figure 6 This is a schematic diagram of the MJPEG encoding and network transmission processing process at the sending end of the present invention;

[0088] Figure 7 This is a flow chart of data processing at the receiving end in the present invention;

[0089] Figure 8 This is a schematic diagram of the MJPEG / PCM data receiving and playing process at the receiving end of the present invention. DETAILED DESCRIPTION

[0090] The present invention provides a data transmission method with dynamic code rate adjustment based on HDMI, which effectively improves the long-distance transmission quality and stability of HDMI signals through congestion control during data transmission, and solves the problem of difficult long-distance transmission wiring.

[0091] When the distance is relatively close, the sender and receiver are connected via an ordinary network cable (within 50 meters of transmission distance), and the two can communicate with each other because they are directly connected and are not easily affected by external interference. When the transmission rate drops due to external interference, the dynamic bit rate adjustment control system reduces the bit rate and the amount of data to ensure low latency and smooth video.

[0092] When the distance is long, such as in different rooms, floors, or buildings, the receiver and transmitter are connected to network infrastructure such as routers and switches via network cables. As long as they are within the same local area network, the transmitter and receiver automatically discover each other through the local area network discovery protocol and establish a connection with the receiver via TCP or UDP, allowing them to send audio, video data, and control information to each other. If the transmission rate drops due to external interference, the dynamic bit rate adjustment control system reduces the bit rate and data volume to ensure low latency and smooth video images.

[0093] Figure 1 Schematic diagram of the hardware structure of the transmitting end and the receiving end of the present invention:

[0094] (1) Sending end

[0095] The transmitter includes an HDMI input interface, a SoC (System on Chip) module, and a network module. The HDMI input interface receives audio and video signals from a source (such as a Blu-ray player, TV, or computer). The SoC module encodes the received audio and video signals using MJPEG for video and Pulse Code Modulation (PCM) for audio. The network module converts the encoded audio and video signals into network data packets and transmits them to the receiver via Ethernet or Wi-Fi, along with control information.

[0096] (2) Receiver

[0097] The receiving end includes an HDMI output interface, a SoC module, and a network module. The network module receives audio and video data packets transmitted from the network, as well as the control information sent. The SoC module decodes the received audio and video data packets and restores them to the original audio and video signals, which are then output to a display device (such as a TV or projector) through the HDMI output interface. The HDMI output interface outputs the decoded audio and video signals to the display device, achieving the ultimate visual and auditory experience.

[0098] There are two ways to connect the sending and receiving ends to the network:

[0099] (1) Direct network connection: The sender and receiver are directly connected via Ethernet cables, such as Figure 2 As shown, the data transmission speed and stability are relatively high, which is suitable for local network environments.

[0100] (2) Connection through network infrastructure: The sender and receiver are connected through switches or routers in the local area network, such as Figure 3 This method is suitable for large-scale, multi-device environments and can effectively support connections between multiple senders and receivers.

[0101] HDMI output devices transmit image and audio signals to the transmitter via an HDMI cable. Upon receiving the HDMI signal, the transmitter encodes the image signal into MJPEG format and the audio signal into PCM format, and transmits the encoded data to the receiver via Ethernet. The receiver decodes the received MJPEG and PCM data and outputs it to the connected display device via the HDMI interface. During data transmission, if network congestion is detected, resulting in increased transmission latency, the transmitter will automatically reduce the bitrate to reduce the data volume, thereby alleviating network load and reducing latency. Conversely, when network conditions are good, the transmitter will increase the bitrate, increasing the data volume and improving image quality.

[0102] like Figure 4 As shown, the processing flow at the sending end is as follows.

[0103] (1) Startup: At the beginning of startup, the hardware and software components of the transmitter are initialized to ensure that they can perform subsequent tasks. This includes the initialization of the HDMI audio and video encoding hardware module and its driver. The hardware module includes an MJPEG video encoder and a PCM audio encoder, which convert the HDMI video and audio signals into MJPEG format (i.e., motion JPEG) data and PCM (pulse code modulation) format data respectively. The generation of MJPEG format data comes from compressing each frame in the video sequence separately using the JPEG format. For example, the image information in the YUV format (such as YUV422) transmitted from HDMI is compressed frame by frame, and then each frame of JPEG data is sequentially spliced ​​together to form an MJPEG stream. HDMI can transmit multi-channel audio (such as LPCM, Dolby Digital, DTS, etc.), where LPCM itself is in PCM format. If the collected audio is in a compressed format (such as Dolby Digital or DTS), it is decoded into PCM format. At the same time, the transmitter network module and its protocol stack are also initialized to be able to send and receive network data.

[0104] (2) Obtaining the sender's IP address: The sender requests the network IP allocation device to assign an IP address to itself through the network interface. If a valid IP address cannot be assigned, the sender will repeat the request, sleeping for 500ms between each re-acquisition, until the IP address is successfully obtained before proceeding to the next step.

[0105] (3) Listen for multicast messages sent by the receiver: Listen for broadcast packets with multicast address 239.0.0.2 and port 9000 to obtain the paired receiver's IP address. Because different IP addresses may be obtained in different networks, the sender does not know the receiver's IP address. Therefore, the LAN discovery protocol is needed to discover the receiver and obtain the receiver's IP address and other information. Multiple identical devices may be running on the same LAN. The sender has stored the serial number of the paired receiver at the factory. By comparing it with the serial number in the received broadcast message, the sender can find the paired receiver.

[0106] (4) Is it the paired receiving end serial number: Compare the serial number in the received multicast message with the serial number of the receiving end to see if they are consistent. If so, proceed to the next step; otherwise, return to step (3) and continue to monitor the multicast messages sent by the receiving end to obtain the paired receiving end.

[0107] (5) Update the receiving end network IP address: store the receiving end IP address.

[0108] (6) Start the HDMI audio and video encoding hardware module and network inside the SOC: start the MJPEG video encoder and PCM audio encoder, set the bit rate and frame rate of the encoded video, as well as the audio encoding format and bit rate.

[0109] (7) Detect HDMI input source signal: If HDMI input signal is detected, proceed to the next step, otherwise wait for the HDMI input source to be connected to the HDMI transmitter.

[0110] (8) Start the MJPEG and PCM encoding, reading, and data sending threads: allocate MJPEG and PCM data buffer memory, initialize the code table for adjusting the video bit rate, and then start the PCM encoding and network sending thread and the MJPEG encoding and data sending thread over the network.

[0111] (9) Loop monitoring of multicast messages sent by the receiving end: monitor broadcast packets with multicast address 239.0.0.2 and port 9000. If the serial number in the received message is the same as its own, update the receiving end IP address and continue to monitor multicast messages.

[0112] The PCM encoding and network sending thread processing process is as follows Figure 5 As shown:

[0113] (1) Establish a TCP network connection with the receiving end: After the thread is started, try to establish a TCP network connection with the receiving end and determine whether the connection is successful. If it fails, close the network connection and try again until the TCP network connection is successfully established.

[0114] (2) PCM data acquisition, encoding, and segmentation: PCM-encoded data is read from the audio encoding driver pre-installed in the SOC of the sending end. The data processing program segments the PCM data into 1400-byte segments and adds a CRC checksum header to each data segment so that the receiving end can verify the integrity of the data. The purpose of designing the segment length to 1400 bytes is to improve network transmission efficiency. This length will not cause caching, packetization, or grouping operations when passing through the network protocol stack. The protocol stack will directly call the network interface to send data, thereby reducing the data processing time at the sending and receiving ends. The CRC value only calculates the data header and does not calculate the 1400-byte PCM data, saving time and CPU usage.

[0115] (3) PCM data transmission: The segmented PCM data is sent to the receiving end through the established TCP connection.

[0116] (4) Determine whether the data is sent successfully: If the sending fails and the TCP connection is not disconnected, discard the current PCM data and re-collect the next frame of data; if the sending fails and the TCP connection is disconnected, return to the first step to re-establish the TCP network connection; if the sending is successful, proceed to the next step.

[0117] (5) Check whether the receiving end IP address is updated: If the receiving end IP address has been updated, close the current network connection and return to step (1) to re-establish a TCP connection with the receiving end. If the transmission is successful, return to step (2) and continue the subsequent PCM data collection, encoding and segmentation.

[0118] The MJPEG encoding and network sending thread processing process is as follows Figure 6 As shown:

[0119] (1) Establish a TCP network connection with the receiving end: After the thread starts, try to establish a TCP network connection with the receiving end. Determine whether the connection is successful. If it fails, close the network connection and try again until the TCP network connection is successfully established.

[0120] (2) MJPEG data acquisition, encoding, and segmentation: The MJPEG encoded data is read from the encoding driver. The data processing program segments the MJPEG data into 1400-byte segments and adds a CRC checksum header to each data segment so that the receiving end can verify the integrity of the data. The purpose of designing the segment length to 1400 bytes is to improve network transmission efficiency. This length does not cause caching, packetization, or grouping operations when passing through the network protocol stack. The protocol stack will directly call the network interface to send data, thereby reducing the data processing time at the sending and receiving ends. The CRC value only calculates the data header and does not calculate the 1400-byte MJPEG data, saving time and CPU usage.

[0121] (3) Data transmission: The segmented MJPEG data is sent to the receiving end through the established TCP connection.

[0122] (4) Determine whether the data is sent successfully: if the sending fails, proceed to the next step; if the sending is successful, proceed to step (8).

[0123] (5) Check the receiving end network: If the receiving end network is disconnected, proceed to the next step, otherwise proceed to step (7).

[0124] (6) Close the network: Close the TCP network connection with the receiving end and go to step (1).

[0125] (7) Adjust the MJPEG encoding code table: discard the MJPEG encoded data, adjust the MJPEG encoding code table, and reduce the MJPEG bit rate.

[0126] (8) Check whether the receiving end IP address is updated: If the receiving end IP address has been updated, go to step (6), otherwise go to the next step.

[0127] (9) Adjusting the video MJPEG encoding bitrate: There are two video encoding resolutions: less than or equal to 1280x720 and greater than 1280x720. The encoding frame rate of the former is 60 frames per second, and the frame rate of the latter is 30 frames per second. The maximum data volume per second is preset to not exceed 11MByte / s. Assume that the current video frame size is X. The video bitrate can be adjusted in two cases:

[0128] ① When the resolution is less than or equal to 1280*720 and the frame rate is 60 frames / s:

[0129] The average data size of each frame is:

[0130] A=11*1024*1024 / 60=192238(bytes / s)

[0131] Assume that the adjustment coefficient λ = 0.2.

[0132] When X>A+0.2*A=192238+0.2*192238=230685, it indicates data overload. The network burden can be reduced by appropriately adjusting the code table to reduce the data volume of each frame.

[0133] When X < A - 0.2 * A = 192238 - 0.2 * 192238 = 153790, it indicates that the data volume is too low. The code table can be appropriately adjusted to increase the data volume of each frame to improve video quality. In this embodiment, the adjustment is performed using 0.2 times the average of addition and subtraction. In actual application, the adjustment should be based on hardware conditions and video quality. The larger the value, the fewer bitrate adjustments are made, and the video encoding quality is stable. However, the adjustment response speed is slow, which may cause network congestion and delayed improvement at low bitrates, affecting video quality. The smaller the value, the more sensitive it is to bitrate fluctuations, and the faster it can respond to overload or low bitrate issues. However, frequent adjustments lead to unstable video encoding quality.

[0134] Alternatively, a rate adjustment scheme based on PID (Proportional-Integral-Derivative) control can be implemented to achieve more accurate rate adjustment through dynamic error feedback. The specific method is as follows:

[0135] The goal of PID control is to stabilize the current frame data volume X near the target average value A. The adjustment amount ΔR is calculated using the following formula:

[0136]

[0137] e(t) = XA (current error): X is the amount of data in the current frame, and A is the target average value, which is 192238 in this case.

[0138] ·K p (Proportional coefficient): directly reflects the impact of the current error. The larger the value, the more radical the adjustment. Here, it is set to 0.1.

[0139] ·K i (Integral coefficient): Eliminate historical accumulated errors (such as long-term deviations), and take 0.01 here;

[0140] ·K d (Differential coefficient): predicts the error change trend and suppresses oscillations. Here it is set to 0.05.

[0141] When the calculated ΔR>0 and is greater than a certain threshold value, the bit rate is increased;

[0142] When the calculated ΔR is less than 0 and its absolute value is greater than a certain threshold, the bit rate is reduced;

[0143] This solution makes the response faster, prevents frequent adjustments, and adjusts K according to actual scenarios (such as network stability and video type). p , K i and K d For example, in a poor network scenario, increase K d Suppress oscillation and increase K for dynamic video p Improve response speed.

[0144] ②When the resolution is greater than 1280*720 and the frame rate is 30 frames / s:

[0145] The average data size of each frame is:

[0146] A=11*1024*1024 / 30=384477(bytes / s)

[0147] Assume that the adjustment coefficient λ = 0.2.

[0148] When X>A+0.2*A=384477+0.2*384477=461372, it indicates data overload. The network burden can be reduced by appropriately adjusting the code table to reduce the data volume of each frame.

[0149] When X<A-0.2*A=384477-0.2*384477=307581, it means that the data volume is too low. The video quality can be improved by properly adjusting the code table to increase the data volume of each frame.

[0150] The amount of data in each frame fluctuates greatly. In order to reduce frequent bit rate adjustments, there should be at least 3 frames between each bit rate adjustment.

[0151] The receiving end processing process is as follows Figure 7 As shown:

[0152] (1) Start: Turn on the power.

[0153] (2) Boot Program Initialization: When the system starts booting, it goes through the initialization phase of hardware and software components to ensure that they can perform subsequent tasks. This includes the initialization of the HDMI audio and video decoding hardware and its driver. This hardware module is responsible for converting audio and video into HDMI signals for display devices. At the same time, the network module and its protocol stack are also initialized to be able to send and receive network data.

[0154] (3) Check whether the hardware configuration is a direct network cable connection solution: Determine whether a direct network cable connection solution is adopted by reading the GPIO L0 pin of the hardware. When it is at a high level, the receiving end and the sending end are in a direct connection solution.

[0155] 3.1 Direct connection solution: The receiving end provides an IP address to the sending end, starts the DHCP server, and listens for IP address allocation requests from the sending end. If an IP address allocation request is received, a valid IP address is allocated and replied to the sending end through the network. The program does not need to wait for the sending end to request an IP address. After starting the DHCP service, the network driver automatically allocates the IP address and replies to the request. After startup, it enters step 4.

[0156] 3.2 In the non-direct connection solution, the router provides the IP address, so polling is performed to obtain the IP address until a valid IP address is obtained, and then proceed to step 4.

[0157] (4) Initialize multicast service: Initialize the multicast service, the multicast address sent is 239.0.0.2, and the port is 9000. Initialize the UI and start the MJPEG / PCM data receiving and playing thread: Initialize the user interface, prompt the user that the application initialization is complete and start the MJPEG / PCM data receiving and playing thread.

[0158] (5) Whether the receiving end and the sending end have established a connection: Check whether the sending end and the receiving end have established a connection. If so, go to step 7; otherwise, go to step 6.

[0159] (6) Display "No Signal" on the UI to prompt the user: prompt the user that the receiving end and the sending end have not yet established a connection, and then go to step 8.

[0160] (7) Hide UI: Hide the UI and display the received video.

[0161] (8) Sleep 10ms: The current thread sleeps for 10ms to reduce CPU usage.

[0162] (9) Whether it has been more than 100ms since the last broadcast of device information: Check whether it has been more than 100ms since the last broadcast of device information. If so, go to step (10), otherwise go back to step (5).

[0163] (10) Broadcast device information: A multicast message is sent through the multicast service created in step 4. The multicast message contains the receiver's serial number, IP address, and other information. When the sender receives a multicast message that matches its own serial number, it can obtain the receiver's IP address from the multicast message. Using this IP address, the sender and receiver can establish a TCP connection and send data to each other. After sending the multicast message, return to step 4.

[0164] The process of receiving and playing MJPEG / PCM data is as follows Figure 8 Shown:

[0165] (1) Start an independent thread to receive and play MJPEG / PCM data.

[0166] (2) Initialize the TCP server and MPEG / PCM playback: Initialize the TCP server and start the playback device of MJPEG / PCM data.

[0167] (3) Waiting for the sender to request a TCP network connection: The system enters a waiting state, waiting for the sender to request a TCP network connection.

[0168] (4) The TCP network connection between the sender and the receiver is successful: If the sender requests a connection and the connection is successful, go to step 5, otherwise go to step 3.

[0169] (5) Receiving data: The receiving end receives the MJPEG and PCM segmented data sent by the sending end.

[0170] (6) Check whether the data is received successfully: Determine whether the network protocol stack returns successfully. If successful, execute step 8; otherwise, execute step 7.

[0171] (7) Sending client whether it is closed: Check the returned error code. If it is "network is closed", execute step 3; otherwise, execute step 5.

[0172] (8) Check whether the CRC of the segmented data packet header is correct and whether the segmented data is continuous: This step checks the CRC of the segmented data header to determine whether a frame of data is continuous. If the CRC is correct and the received data segment is continuous, execute step 10, otherwise execute step 9.

[0173] (9) Discard MJPEG / PCM data: Discard the received audio and video data, release the occupied memory, and then execute step 5.

[0174] (10) Storing segmented data packets: The received segmented data includes audio data and video data, and the segmented data is stored in the audio cache queue and the video cache queue.

[0175] (11) Whether the frame data can be assembled: Check whether the sequence numbers of the data segments in the video cache queue and the audio cache queue are continuous, and whether the total number of segments received is equal to the total number of segments of the current frame saved in the segment header, which means that a complete frame of data has been received. If it is a complete frame of data, execute step 12, otherwise execute step 5.

[0176] (12) Play MJPEG / PCM data: Write a frame of audio data or video data to the audio playback driver or video playback driver, convert the audio and video data into HDMI signals, output them to the display device, and then execute step 5.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for dynamic bit rate adjustment data transmission based on HDMI, comprising the following steps: Step 1: Initialize the HDMI audio and video encoding hardware module and its driver at the sending end, which includes an MJPEG video encoder and a PCM audio encoder; the sending end requests an IP address, and proceeds to the next step after successfully obtaining the IP address; Step 2: Monitor the multicast message sent by the receiving end, compare the sequence number in the sending end and the multicast message, obtain the paired receiving end and store its IP address; If the paired receiver fails to be found, continue to monitor the multicast message; Step 3. Start the MJPEG video encoder and PCM audio encoder, set the video encoding bit rate and frame rate, as well as the audio encoding format and bit rate; Step 4: Detect the HDMI input source signal. If an HDMI input signal is detected, proceed to the next step. Otherwise, wait for the HDMI input source to be connected to the HDMI transmitter. Step 5: Allocate MJPEG and PCM data buffer memory, initialize the code table for adjusting the video bit rate, and then start PCM encoding and network transmission processing, and MJPEG encoding and network transmission processing respectively; Step 6: Circularly monitor the multicast messages sent by the receiving end. If the sequence number in the received message is the same as the sender's own, update the receiving end IP address and continue to monitor the multicast messages.

2. The method for dynamic bit rate adjustment data transmission based on HDMI according to claim 1, wherein the PCM encoding and network transmission processing in step 5 comprises: (1) Establish a TCP network connection with the receiving end and determine whether the connection is successful. If it fails, close the network connection and try again until the TCP network connection is successfully established. (2) Read the PCM data, segment it into segments according to the predetermined number of bytes, and add a CRC checksum to each data segment; (3) Send the segmented PCM data to the receiving end through the established TCP connection; (4) Determine whether the data is sent successfully. If the sending fails and the TCP connection is not disconnected, discard the current PCM data and re-collect the next frame of data; If the sending fails and the TCP connection has been disconnected, go back to step (1) to re-establish the TCP network connection; if the sending succeeds, go to the next step; (5) Check whether the receiving end IP address has been updated. If the receiving end IP address has been updated, close the current network connection and return to step (1) to re-establish a TCP connection with the receiving end. If the transmission is successful, return to step (2) and continue the subsequent PCM data collection, encoding and segmentation.

3. A method for data transmission with dynamic bit rate adjustment based on HDMI as claimed in claim 2, wherein the predetermined number of bytes is 1400 bytes.

4. The method for dynamic bit rate adjustment data transmission based on HDMI as claimed in claim 1 , wherein the MJPEG encoding and network transmission processing in step 5 comprises: (1) Establish a TCP network connection with the receiving end and determine whether the connection is successful. If it fails, close the network connection and try again until the TCP network connection is successfully established. (2) Read the MJEPG data, segment it into segments according to the predetermined number of bytes, and add a CRC checksum to each data segment; (3) Sending the segmented MJPEG data to the receiving end through the established TCP connection; (4) Determine whether the data is sent successfully. If the sending fails, proceed to the next step; if the sending succeeds, proceed to step (8); (5) Check the receiving end network. If it is disconnected from the receiving end network, proceed to the next step; otherwise, proceed to step (7). (6) Close the network, close the TCP network connection with the receiving end, and go to step (1); (7) Discard MJPEG encoded data, adjust the MJPEG encoding code table, and reduce the MJPEG bit rate; (8) Check whether the receiving end IP address is updated. If the receiving end IP address has been updated, go to step (6), otherwise go to the next step; (9) Adjust the video MJPEG encoding bit rate: Calculate the average data size A per frame based on the video encoding resolution, frame rate, and the preset maximum data volume per second. When X>A+λ*A, it indicates that the data volume is overloaded, and the code table should be adjusted to reduce the data volume per frame to reduce the network burden. When X<A-λ*A, it indicates that the data volume is too low, and the code table should be adjusted to increase the data volume per frame to improve the video quality. Where X is the current video frame size read, and λ is the adjustment coefficient. 5 . The method for data transmission with dynamic bit rate adjustment based on HDMI as claimed in claim 4 , wherein the predetermined number of bytes is 1400 bytes and the adjustment coefficient λ is 0.

2.

6. The method for dynamic bitrate adjustment data transmission based on HDMI as claimed in claim 4, wherein the bitrate of the video MJPEG encoding is adjusted based on proportional, integral, and differential PID control. The goal of the PID control is to stabilize the current frame data volume X near the target average value A. The adjustment amount ΔR is calculated by the following formula: in, Current error e(t) = XA, where X is the amount of data in the current frame and A is the target average value; Proportional coefficient K p Indicates the impact of the current error, the larger the error, the more radical the adjustment; Integration coefficient K i Used to eliminate historical accumulated errors; Differential coefficient K d Used to predict error change trends and suppress oscillations; When the calculated ΔR>0 and is greater than a certain threshold value, the bit rate is increased; When the calculated ΔR is less than 0 and its absolute value is greater than a certain threshold, the bit rate is reduced.

7. A method for dynamic bit rate adjustment data transmission based on HDMI as claimed in claim 6, wherein K p The value is 0.1, K i The value is 0.01, K d The value is 0.

05.

8. The method for data transmission based on HDMI with dynamic bit rate adjustment according to claim 1, further comprising a receiving end processing step: (1) Power on and start the program initialization; (2) Check whether the hardware configuration is a direct network connection scheme: Determine whether a direct network connection scheme is adopted by reading the hardware's GPIO L0 pin. When it is high, the receiving end and the sending end are directly connected, and the receiving end's IP address is determined. In a non-direct connection scheme, the router provides an IP address, and the IP address is obtained by polling until a valid IP address is obtained and then proceed to the next step; (3) Initialize the multicast service, initialize the user interface UI, prompt the user that the application initialization is complete, and start receiving and playing MJPEG / PCM data; (4) Check whether the sending end and the receiving end have established a connection. If so, proceed to step (6); otherwise, proceed to step (5); (5) Display "No Signal" on the UI to prompt the user, indicating that the receiving end and the sending end have not yet established a connection, and then proceed to step (7); (6) Hide the UI and display the received video; (7) The current thread sleeps to reduce CPU usage; (8) Check whether the time from the last broadcast of the device information has exceeded the predetermined time. If it has exceeded, go to step (9); otherwise, return to step (4); (9) Send a multicast message through the multicast service created in step (3). When the sender receives a multicast message that matches its own sequence number, it obtains the IP address of the receiver from the multicast message. The sender and receiver establish a TCP connection and send data to each other. After sending the multicast message, return to step (3).

9. The method for transmitting data with dynamic bit rate adjustment based on HDMI as claimed in claim 1, further comprising the steps of receiving and playing the MJPEG data and PCM data at the receiving end: (1) Start the processing thread for receiving and playing MJPEG and PCM data; (2) Initialize the TCP server and start the playback device of MJPEG and PCM data; (3) The system enters a waiting state, waiting for the sender to request a TCP network connection; (4) The TCP network connection between the sender and the receiver is successful. If the sender requests a connection and the connection is successful, proceed to step (5); otherwise, proceed to step (3). (5) The receiving end receives the MJPEG and PCM segmented data sent by the sending end; (6) Check whether the data is received successfully and determine whether the network protocol stack returns successfully. If successful, execute step (8); otherwise, execute step (7); (7) Check whether the sending client is closed and the returned error code. If it is "network is closed", execute step (3); otherwise, execute step (5); (8) Check whether the CRC of the segmented data packet header is correct and whether the segmented data is continuous. If the CRC is correct and the received data segment is continuous, execute step (10); otherwise, execute step (9); (9) Discard the received audio and video data, release the occupied memory, and execute step (5). (10) The received segmented data includes audio data and video data, and the segmented data is stored in the audio cache queue and the video cache queue; (11) Check whether the sequence numbers of the data segments in the video cache queue and the audio cache queue are continuous, and whether the total number of received segments is equal to the total number of segments of the current frame stored in the segment header. If so, it indicates that a complete frame of data has been received, and step (12) is executed; otherwise, step (5) is executed; (12) Playing MJPEG and PCM data: writing a frame of audio data or video data into the audio playback driver or video playback driver, and converting the audio and video data into HDMI signals, outputting them to the display device, and then executing step (5).

Images