Stable display method based on UDP (User Datagram Protocol) in real-time image transmission process

Abstract

The invention relates to a stable display method based on a UDP (User Datagram Protocol) in a real-time image transmission process. The stable display method comprises the following steps of: constructing a UI (User Interface) basic class in a Qt framework; constructing three thread objects in the basic class, namely data receiving, data processing and image processing; in the data receiving thread, image data monitoring is carried out in a UDP network port channel in real time and is transmitted to the data analysis thread, and circulation is carried out; meanwhile, adding a dynamic buffer setting algorithm in a slot function; starting a data processing thread, performing analysis processing on the received data packet, transmitting complete image data to an image processing thread, and continuing to repeat the operation; the image processing thread is responsible for transmitting the image data to an image display control of the UI interface for display, and the operation is continuously repeated; and observing an algorithm execution effect, and evaluating a resource utilization rate and image integrity. According to the invention, the problem of unstable image display in the real-time image transmission process of the laser communication UDP can be solved, and stable display of the image with minimum resources is ensured.

Description

Technical Field

[0001] The present invention relates to a display method, specifically a stable display method during UDP real-time image transmission. Background Art

[0002] In the field of laser communication, the pre-steps of laser communication include three stages: aiming, acquisition, and tracking. Among them, acquisition and tracking require obtaining the centroid position of the laser spot in the laser spot image sent by the peer device for subsequent series of operations. In the acquisition stage, quickly and accurately capturing the spot and reducing the time required for scanning and acquisition are important objectives of the acquisition stage.

[0003] The laser spot is captured by a ccd camera and then the image data is restored on the host computer through network transmission, and subsequent logical judgments and response operations are performed on the displayed image. Therefore, the integrity and stability of the image can improve the efficiency of the laser spot capture process.

[0004] However, the real-time image transmission based on UDP is prone to affecting the imaging effect of the picture due to its own mechanism, and it is easy to have image frame loss, image jamming, and even directly cause the software to crash. The unreliability of UDP transmission makes it difficult to ensure that the rates at both ends of data sending and receiving are consistent. In the case of high frames, the rate mismatch phenomenon will be more serious, resulting in a large amount of image data loss during the real-time data transmission process, thus leading to the instability of image display. Summary of the Invention

[0005] The present invention aims to solve the problem of unstable image display caused by the rate mismatch between the data sending and receiving ends during the UDP real-time image transmission in laser communication, and at the same time ensure that the image is stably displayed with the least resources.

[0006] The present invention proposes a stable display method during UDP real-time image transmission, including the following steps:

[0007] Step 1), construct the basic class of the UI interface in the Qt framework to implement thread management and image display;

[0008] Step 2), construct three thread objects in the basic class to implement the data processing flow, where the functions of the three threads are data reception, data processing, and image processing;

[0009] Step 3), in the data reception thread, continuously monitor the image data in the UDP network port channel in real time, and then read the image data packets stored in the UDP reception buffer through the data reception processing slot function and pass them to the data parsing thread, and repeat the process; at the same time, add a dynamic buffer setting algorithm in the slot function to change the dynamic buffer size in real time to ensure that all data of the image can be completely received;

[0010] Step 4), start a data processing thread to parse and process the received data packet, obtain the complete data of an image, and transmit the complete image data to the image processing thread through a signal, and continue to repeat the above operations;

[0011] Step 5), the image processing thread is responsible for converting the unpacked and restored image data into an image processing format, and after completion, transmit the image data to the image display control of the UI interface for display, and continue to repeat the above operations;

[0012] Step 6), observe the execution effect of the algorithm and evaluate the resource utilization rate and image integrity;

[0013] After adding the algorithm, after going through steps S3 - S5, the basic class obtains the image format that has been processed by the image processing thread, and loads it into the image display control in the Ui interface to display its image, and observe the view display situation.

[0014] As a further improvement of the present invention, the specific steps of the algorithm in step 3) are as follows:

[0015] The first step is to judge in the data receiving thread whether all data packets of an image have been completely received; if not, obtain the current receiving buffer size bufferNum, and if completely received, skip;

[0016] The second step is to obtain the ratio V of the reading rate to the receiving rate of the data in the receiving buffer at this rate through algorithm formula (1 - 1);

[0017] The algorithm formula is:

[0018]

[0019] Where sum(rec) is the number of data packets of an image received in the data receiving thread, and sum(rec) - bufferNum is the number of data packets received in the receiving buffer when the receiving buffer is full; where sum(rec) is the total number of packets read by the receiving buffer at this time. By comparing the number of data packets received in the receiving buffer with the total number of packets read, the data receiving and transmitting rate ratio V of the receiving buffer at this time can be obtained;

[0020] The third step is to obtain the minimum resource receiving buffer size bufferNum(small) according to algorithm formula (1 - 2) through the receiving and transmitting rate ratio V of the buffer;

[0021] The algorithm formula is:

[0022] bufferNum(small) = (1 - V) * sum(all) (1 - 2)

[0023] where sum(all) is the total number of packets of an image data;

[0024] Step 4, modify the function through the UDP underlying interface;

[0025] setUdpReceiveBufferSize(m_UDPSocket, desiredBufferSize) sets the size of the receive buffer;

[0026] where m_UDPSocket is the socket corresponding to the modified UDP buffer, and desiredBufferSize is the size of the receive buffer to be set.

[0027] As a further improvement of the present invention, the method of real-time monitoring of image data in the UDP network port channel in step 3) is as follows: by using the UDP socket as a private member variable of the data receiving thread, calling the connect connection function to connect the data monitoring signal ready of the UDP socket and the data receiving processing slot function in the thread; once data arrives in the network port channel, the ready signal will be triggered, and then the slot function will be called to perform corresponding operations.

[0028] As a further improvement of the present invention, there are the following two steps to read the image data stored in the UDP receive buffer in the slot function in step 3):

[0029] Step 1: Use a while loop in the slot function to call the hasPendingDatagrams function of UDP as a judgment condition;

[0030] Among them, the hasPendingDatagrams function of UDP is used to check whether there are datagrams available for processing in the UDP receive buffer;

[0031] Step 2: When data is stored in the UDP receive buffer, call the readDatagram function of UDP to receive and read it;

[0032] Among them, the readDatagram function of UDP is a member function of the QUdpSocket class in the Qt framework, which is used to read the arrived UDP datagrams;

[0033] Among them, the image packet data is passed to the data parsing thread by sending a signal through the emit keyword.

[0034] As a further improvement of the present invention, the process of parsing and processing the data packet in step 4) is as follows: judge the received data packet frame header and the sequence number in the packet through the communication protocol of the image; merge the judged data into a complete image packet and pass it to the image processing thread through a signal.

[0035] As a further improvement of the present invention, the judgment basis is as follows:

[0036] Only when the communication protocol standards are met will the data packet be parsed and judged;

[0037] Judging the sequence number in the packet is to determine whether the data in the packet is for the same picture. When the sequence number in the packet is larger than that of the previous data packet, it can be determined that the data is for the same picture; if the sequence number in the packet starts from 0 or is smaller than that of the previous data packet, it can be considered that the data is not for the same picture;

[0038] The method for judging a complete image packet is to record the sequence number in the packet of the collected image data. If the sequence number starts from 0 again, it is determined that the image data of the previous image has been completely collected;

[0039] The merged complete image packet is passed to the image processing thread, and the steps are the same as the method for the data receiving thread to pass data in S3.

[0040] As a further improvement of the present invention, the specific operation of step 5) to start the image processing thread is as follows: when the data parsing thread sends a signal to transfer the image packet, the basic class will immediately call the start function of the image processing thread upon receiving this signal, thereby starting the image processing thread; after processing, the image processing thread passes the image data to the image display control on the UI interface for display, and then continues to repeat the above operations.

[0041] As a further improvement of the present invention, the step of converting the image packet into an image display format in step 5) is as follows:

[0042] Step 1: Construct an image object for storing a grayscale image in the image processing thread. Among them, the function for constructing this object is:

[0043] QImage* image = new QImage(width, height, QImage::Format_Grayscale8)

[0044] where width is the actual width of the image;

[0045] height is the actual height of the image;

[0046] QImage::Format_Grayscale8 is the format for constructing a grayscale image that stores only 1 bit depth;

[0047] Step 2: Read the received image packet data in a loop and store it in the image object.

[0048] As a further improvement of the present invention, the specific operation for judging the integrity of the picture display in step 6) is as follows: The picture is displayed in real time through the image display control for half an hour, and the view display situation is observed. During this period, there are no white bars, lags, or noise phenomena.

[0049] The beneficial effects of the present invention are as follows:

[0050] Through improvements in structure and algorithm, the present invention utilizes the ratio of the sending and receiving rates of the UDP receive buffer and the camera sending frame rate to obtain the size of the receive buffer under the minimum resources, and can obtain it in real time, improving the stability of image display and the test efficiency, so as to quickly help users measure the receive buffer boundary under this frame rate and network conditions;

[0051] During the real-time display process, open the task manager to view the resource utilization rate of the display process, including CPU utilization rate and memory occupancy rate. Generally, it only occupies 20%-25% of the total. Compared with the 40%-50% occupied without adding the algorithm, the resource consumption is reduced by half. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the main process of the present invention.

[0053] Figure 2 It is a schematic diagram of the test case generation of the present invention.

[0054] Figure 3 It is a schematic diagram of the logic of the use case generation for the experience library of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0055] The present invention will be further described below in conjunction with the embodiments in the drawings:

[0056] As shown in the figure, a method for stable display during UDP real-time image transmission includes the following steps:

[0057] Step 1), construct the basic UI interface class in the Qt framework to implement thread management and image display;

[0058] Step 2), construct three thread objects in the basic class to implement the data processing flow, where the functions of the three threads are data reception, data processing, and image processing;

[0059] Step 3), in the data reception thread, continuously monitor the image data in the UDP network port channel in real time, and then read the image data packets stored in the UDP receive buffer through the data reception and processing slot function, and transfer them to the data parsing thread, repeating this process; at the same time, add a dynamic buffer setting algorithm to the slot function to change the dynamic buffer size in real time to ensure that all data of the image can be completely received;

[0060] The specific steps of the algorithm in step 3) are as follows:

[0061] First, in the data receiving thread, determine whether all data packets of an image have been completely received; if not, obtain the current size of the receiving buffer bufferNum, and if completely received, skip;

[0062] Second, obtain the ratio V of the reading rate to the receiving rate of the data in the receiving buffer at this rate through algorithm formula (1-1);

[0063] The algorithm formula is:

[0064]

[0065] Where sum(rec) is the number of data packets for receiving an image in the data receiving thread, and sum(rec)-bufferNum is the number of data packets received in the receiving buffer when the receiving buffer is full; where sum(rec) is the total number of packets read by the receiving buffer at this time. By comparing the number of data packets received in the receiving buffer with the total number of packets read, the data transceiver rate ratio V of the receiving buffer at this time can be obtained;

[0066] Third, through the transceiver rate ratio V of the buffer, the size bufferNum(small) of the receiving buffer under the minimum resources can be obtained according to algorithm formula (1-2);

[0067] The algorithm formula is:

[0068] bufferNum(small)=(1-V)*sum(all) (1-2)

[0069] Where sum(all) is the total number of packets of an image data;

[0070] Fourth, modify the function through the udp underlying interface;

[0071] setUdpReceiveBufferSize(m_UDPSocket,desiredBufferSize) sets the size of the receiving buffer;

[0072] Where m_UDPSocket is the socket corresponding to the modified udp buffer, and desiredBufferSize is the size of the receiving buffer to be set;

[0073] The method of real-time image data monitoring in the step 3) is as follows: by taking the UDP socket as a private member variable of the data receiving thread, calling the connect function to connect the data monitoring signal ready of the UDP socket to the data receiving and processing slot function in the thread; once data arrives at the network port channel, the ready signal will be triggered, and then the slot function will be called to perform corresponding operations.

[0074] There are the following two steps to read the image data stored in the UDP receiving buffer in the slot function in the step 3):

[0075] Step 1: Use a while loop in the slot function to call the hasPendingDatagrams function of UDP as a judgment condition;

[0076] Among them, the hasPendingDatagrams function of UDP is used to check whether there are datagrams available for processing in the UDP receiving buffer;

[0077] Step 2: When data is stored in the UDP receiving buffer, call the readDatagram function of UDP to receive and read it;

[0078] Among them, the readDatagram function of UDP is a member function of the QUdpSocket class in the Qt framework, which is used to read the arriving UDP datagrams;

[0079] Among them, the image packet data is passed to the data parsing thread by sending a signal through the emit keyword.

[0080] In step 4), start a data processing thread to parse and process the received data packets, obtain the complete data of an image, and pass the complete image data to the image processing thread through a signal, and continue to repeat the above operations;

[0081] The process of data packet parsing and processing in the step 4) is as follows: judge the received data packet frame header and the sequence number in the packet through the communication protocol of the image; merge the judged data into a complete image packet and pass it to the image processing thread through a signal;

[0082] The judgment basis is as follows:

[0083] Only when it meets the standards formulated by the communication protocol will the data packet be parsed and judged;

[0084] The judgment of the sequence number in the packet is to determine whether it is the data in the packet of the same picture. When the sequence number in the packet is larger than the previous data packet, it can be considered as the data in the same picture; if the sequence number in the packet starts from 0 or is smaller than the previous data packet, it can be considered as not the data in the same picture;

[0085] The method for judging the complete image packet is to record the packet sequence number in the collected image data. If the sequence number starts from 0 again, it is determined that the image data of the previous image has been completely collected;

[0086] Transfer the merged complete image packet to the image processing thread, and the steps are the same as those of the data receiving thread in S3 for transferring data.

[0087] In step 5), the image processing thread is responsible for converting the unpacked and restored image data into the format for image processing. After completion, the image data is transferred to the image display control on the UI interface for display, and the above operations are continued to be repeated;

[0088] The specific operation to start the image processing thread in step 5) is as follows: when the data parsing thread sends a signal to transfer the image packet, the base class will immediately call the start function of the image processing thread when receiving this signal, thus starting the image processing thread; after processing, the image processing thread transfers the image data to the image display control on the UI interface for display, and then continues to repeat the above operations;

[0089] The steps to convert the image packet into the image display format in step 5) are as follows:

[0090] Step 1: Construct an image object for storing grayscale images in the image processing thread. Among them, the function to construct this object is:

[0091] QImage* image = new QImage(width, height, QImage::Format_Grayscale8)

[0092] where width is the actual width of the image;

[0093] height is the actual height of the image;

[0094] QImage::Format_Grayscale8 is the format for constructing a grayscale image that only stores 1 bit depth;

[0095] Step 2: Loop to read the received image packet data and store it in the image object.

[0096] In step 6), observe the execution effect of the algorithm and evaluate the resource utilization rate and image integrity;

[0097] After adding the algorithm, after going through steps S3 - S5, the base class obtains the image format that has been processed by the image processing thread and loads it into the image display control on the Ui interface to display its image, and observe the view display situation.

[0098] The specific operation for judging the integrity of the picture display in step 6) is as follows: The picture is displayed in real time through the image display control for half an hour, and the view display situation is observed. During this period, there are no phenomena such as white bars, lags, and noises.

Claims

1. A stable display method based on UDP real-time image transmission process, characterized in that: It includes the following steps: Step 1), build the basic class of UI interface in Qt framework to realize thread management and image display; Step 2), construct three thread objects in the basic class to implement the data processing flow, where the functions of the three threads are data reception, data processing, and image processing; Step 3), in the data receiving thread, the image data is monitored in real time on the UDP network port channel, and then the image data packet stored in the UDP receiving buffer is read through the data receiving processing slot function, and passed to the data parsing thread, and the cycle is repeated; At the same time, a dynamic buffer setting algorithm is added to the slot function to change the dynamic buffer size in real time to ensure that all image data can be fully received; Step 4), start the data processing thread, parse and process the received data packets, obtain the complete data of an image, and pass the complete image data to the image processing thread through a signal, and continue to repeat the above operations; Step 5), the image processing thread is responsible for converting the packaged and restored image data into an image processing format, and after completion, the image data is passed to the image display control of the UI interface for display, and the above operations are repeated; Step 6), observe the algorithm execution effect and evaluate resource utilization and image integrity; After adding the algorithm, go through steps S3-S5, the basic class obtains the image format that has been processed by the image processing thread, and loads it into the image display control in the Ui interface to display its image, and observe the view display.

2. A stable display method in a real-time image transmission process based on UDP as claimed in claim 1, characterized in that: The specific steps of the algorithm in step 3) are: The first step is to determine whether all data packets of an image are completely received in the data receiving thread; if not, obtain the current receiving buffer size bufferNum, and if it is completely received, skip; The second step is to obtain the ratio V of the reading rate and the receiving rate of the receiving buffer data at this rate through the algorithm formula (1-1); The algorithm formula is: Where sum(rec) is the number of data packets received in the data receiving thread for one image, where sum(rec)-bufferNum is the number of data packets received in the receiving buffer when the receiving buffer is full; where sum(rec) is the total number of packets read in the receiving buffer at this time, and by comparing the number of data packets received in the receiving buffer with the total number of packets read, the data sending and receiving rate ratio V of the receiving buffer at this time can be obtained; The third step is to use the buffer's send and receive rate ratio V and the algorithm formula (1-2) to obtain the receive buffer size bufferNum(small) under the minimum resource. The algorithm formula is: bufferNum(small)=(1-V)*sum(all) (1-2) Where sum(all) is the total number of packets of an image data; The fourth step is to modify the function through the udp underlying interface; setUdpReceiveBufferSize(m_UDPSocket, desiredBufferSize) sets the size of the receive buffer; Where m_UDPSocket is the socket corresponding to the modified udp buffer, and desiredBufferSize is the receiving buffer size to be set.

3. A stable display method in a real-time image transmission process based on UDP as claimed in claim 2, characterized in that: The method of monitoring image data in real time on the UDP network port channel in the step 3) is: by using the UDP socket as a private member variable of the data receiving thread, calling the connect function, connecting the data monitoring signal ready of the UDP socket with the data receiving processing slot function in the thread; once data arrives at the network port channel, the ready signal will be triggered, and then the slot function will be called to perform corresponding operations.

4. The stable display method in the real-time image transmission process based on UDP as claimed in claim 2, characterized in that: In the step 3), the image data stored in the UDP receiving buffer is read in the slot function, and there are the following two steps: Step 1: Use a while loop in the slot function to call the hasPendingDatagrams function of UDP as a judgment condition; Among them, the UDP hasPendingDatagrams function is used to check whether there are datagrams available for processing in the UDP receive buffer; Step 2: When data is stored in the UDP receiving buffer, call the UDP readDatagram function to receive and read it; Among them, the UDP readDatagram function is a member function of the QUdpSocket class in the Qt framework, which is used to read the arriving UDP datagram; Among them, the image data packet data is passed to the data parsing thread by calling the emit keyword to send a signal.

5. The stable display method in the process of real-time image transmission based on UDP as claimed in claim 1, characterized in that: The data packet parsing process of step 4) is as follows: judging the frame header of the received data packet and the serial number in the packet through the image communication protocol; merging the judged data into a complete image packet and transmitting it to the image processing thread through a signal.

6. A stable display method in a real-time image transmission process based on UDP as claimed in claim 5, characterized in that: The judgment is based on: Data packets will only be parsed and judged if they meet the standards set by the communication protocol; The purpose of judging the sequence number in the packet is to determine whether it is the data in the packet of the same picture. When the sequence number in the packet is larger than that of the previous data packet, it can be considered as the same picture data; if the sequence number in the packet starts from 0, or is smaller than that of the previous data packet, it can be considered not to be the same picture data; The method for determining a complete image package is to record the serial number in the package of the collected image data. If the serial number starts again from 0, it is determined that the image data of the previous image has been collected completely. The merged complete image package is passed to the image processing thread, and the steps are consistent with the data passing method of the data receiving thread in S3.

7. The stable display method in the process of real-time image transmission based on UDP as claimed in claim 1, characterized in that: The specific operation of starting the image processing thread in step 5) is: when the data parsing thread sends a signal to transmit the image package, the basic class will immediately call the start function of the image processing thread upon receiving the signal, thereby starting the image processing thread; after the processing is completed, the image processing thread will pass the image data to the image display control of the UI interface for display, and then continue to repeat the above operations.

8. The stable display method in the process of real-time image transmission based on UDP as claimed in claim 1, characterized in that: The step of converting the image package into an image display format in step 5) is as follows: Step 1: Construct an image object that stores the grayscale image in the image processing thread, where the function to construct the object is: QImage*image=new QImage(width,height,QImage::Format_Grayscale8) Where width is the actual width of the image; height is the actual height of the image; QImage::Format_Grayscale8 is a grayscale image format that only stores 1 bit depth; Step 2: Loop through the received image packet data and store it in the image object.

9. The stable display method in the process of real-time image transmission based on UDP as claimed in claim 1, characterized in that: The specific operation of judging the integrity of the image display in step 6) is: display the image in real time for half an hour through the image display control, and observe the view display status to ensure that no white bars, freezes, or noises appear during the period.

Images