Radar photoelectric comprehensive display method and system based on GPU and CPU

Abstract

The invention discloses a radar photoelectric comprehensive display method and system based on a GPU and a CPU, and belongs to the technical field of radar photoelectric comprehensive display. The method comprises the following steps: firstly, calling a GPU to create a radar video hard layer and a photoelectric video hard layer, then calling a CPU to preprocess radar video data, and analyzing photoelectric video data; then, a GPU is called to read radar video data and draw a radar image, and the radar image is displayed on a radar video hard layer through an API of the hard layer; a GPU is called to read photoelectric video data and carry out video decoding; displaying the decoded photoelectric video data on a photoelectric video hard layer through an API (Application Program Interface) of the hard layer; and finally, calling the GPU to adjust the windows of the radar video hard layer and the photoelectric video hard layer. According to the method, the efficient computing capacity of the CPU and the multi-channel concurrent processing capacity of the GPU are reasonably utilized, the hardware cost of a radar photoelectric comprehensive processing scheme is effectively reduced in a GPU + CPU matching mode, and the iteration flexibility is improved.

Description

Technical Field

[0001] The present application belongs to the field of radar optoelectronic integrated display technology, and more specifically, to a radar optoelectronic integrated display method and system based on GPU and CPU. Background Art

[0002] The radar optoelectronic integrated display system is an advanced integrated display system that integrates radar and optoelectronic technologies. It combines radar detection and optoelectronic imaging to obtain real-time data, uses high-performance computing systems to perform information fusion and visualization, and displays the processed information in the form of images, thus achieving all-round and multi-angle monitoring of the environment.

[0003] There are two ways to implement the existing radar optoelectronic integrated processing and display system. One is to use FPGA or ARM platform to realize data processing and integrated display, and the other is to realize radar optoelectronic data processing and integrated display by CPU processing and rendering based on graphics processing frameworks such as Qt or MFC. The hardware cost of the first implementation method is too high and the iteration flexibility is low. The second implementation method occupies a large amount of CPU resources and cannot cope with the processing of complex massive graphics and image data, which is not conducive to the realization of multi-source and multi-channel radar optoelectronic integrated display. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present application provides a radar optoelectronic integrated display method and system based on GPU and CPU, which aims to solve the technical problem of excessively high hardware cost of the existing radar optoelectronic integrated processing and display system.

[0005] To achieve the above objectives, in a first aspect, the present application provides a radar optoelectronic integrated display method based on a GPU and a CPU, comprising:

[0006] S1. Acquire radar video data and optoelectronic video data;

[0007] S2, calling GPU to create radar video hard layer and optoelectronic video hard layer;

[0008] S3, calling the CPU to pre-process the radar video data and analyze the optoelectronic video data;

[0009] S4, calling the GPU to read the preprocessed radar video data, and drawing a radar image based on the radar video data, and displaying the radar image on the corresponding radar video hard layer through the hard layer API;

[0010] S5, calling the GPU to read the parsed optoelectronic video data, and performing video decoding on the optoelectronic video data based on the video decoding API; displaying the decoded optoelectronic video data on the corresponding optoelectronic video hard layer through the hard layer API;

[0011] S6. Calling the GPU to adjust the windows of the radar video hard layer and the optoelectronic video hard layer.

[0012] Preferably, in step S1, the radar video data includes radar return data, radar azimuth data and radar main pulse data; and the optoelectronic video data includes video data in various video compression formats.

[0013] Preferably, in step S2, a radar video hard layer or an optoelectronic video hard layer is created for each channel of radar video data or optoelectronic video data through the API of the GPU, the window size and position data are obtained from the configuration file in each channel of radar video data or optoelectronic video data, and the window size and position of the corresponding radar video hard layer or optoelectronic video hard layer are adjusted according to the window size and position data.

[0014] Preferably, in step S3, the radar video data is preprocessed, specifically: calling the CPU to decompress the radar video data, and storing the decompressed radar video data into the radar data ring buffer.

[0015] Preferably, in step S3, the optoelectronic video data is parsed, specifically: calling the CPU to read the optoelectronic video RTSP stream information from the configuration file of the optoelectronic video data, using the multimedia stream transmission library to perform parsing operations including entropy decoding and reordering on the optoelectronic video data, and storing the parsed optoelectronic video data in the optoelectronic data buffer.

[0016] Preferably, the step S4 specifically includes the following sub-steps:

[0017] S41, calling the GPU to read radar video data in batches from the radar data ring buffer into the GPU video memory through DMA;

[0018] S42, using OpenGL to draw the radar video data in the video memory into a radar image;

[0019] S43. Display the radar image on the corresponding radar video hard layer through the hard layer API.

[0020] Preferably, the step S5 specifically includes the following sub-steps:

[0021] S51, calling the GPU to read the optoelectronic video data in batches from the optoelectronic data buffer into the GPU video memory;

[0022] S52, performing decoding operations including IQ inverse quantization, IDCT discrete cosine inverse transform, intra-frame prediction, inter-frame prediction and loop filtering on the optoelectronic video data based on the video decoding API;

[0023] S53. Display the decoded optoelectronic video data on the corresponding optoelectronic video hard layer through the hard layer API.

[0024] Preferably, the step S6 specifically comprises: adjusting the windows of the radar video hard layer and the optoelectronic video hard layer through the API of the GPU, wherein the adjustment comprises: window movement, window scaling, window fusion and overlaying, and changing window transparency.

[0025] In a second aspect, the present application provides a radar optoelectronic integrated display system based on GPU and CPU, comprising:

[0026] A video data acquisition module, used to acquire radar video data and optoelectronic video data;

[0027] The hard layer generation module is used to call the GPU to create radar video hard layers and optoelectronic video hard layers;

[0028] The video data preprocessing module is used to call the CPU to preprocess the radar video data and analyze the optoelectronic video data;

[0029] A radar video display module is used to call the GPU to read the preprocessed radar video data, draw a radar image based on the radar video data, and display the radar image on the corresponding radar video hard layer through the hard layer API;

[0030] The optoelectronic video display module is used to call the GPU to read the parsed optoelectronic video data, and perform video decoding on the optoelectronic video data based on the video decoding API; and display the decoded optoelectronic video data on the corresponding optoelectronic video hard layer through the hard layer API;

[0031] The display adjustment module is used to call the GPU to adjust the windows of the radar video hard layer and the optoelectronic video hard layer.

[0032] In a third aspect, the present application provides an electronic device, comprising: a memory for storing programs; a processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is used to execute any possible implementation method of the first aspect.

[0033] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0034] (1) The present application utilizes the high computing performance of the CPU to realize preprocessing and parsing of radar video data and optoelectronic video data, and utilizes the hardware acceleration capability of the GPU for multi-channel concurrent processing to realize the display of multi-channel radar video data and optoelectronic video data. Thus, the radar optoelectronic integrated display task of multi-channel radar data and multi-channel optoelectronic data is completed through the cooperation of the CPU + GPU. Compared with the use of FPGA to realize multi-channel video data processing and integrated display, the present application can effectively reduce the hardware cost of the radar optoelectronic integrated processing solution and improve the flexibility of iteration.

[0035] (2) This application utilizes the hard layer function provided by the GPU to enable the radar optoelectronic display position and size to be adjusted arbitrarily according to user needs, and the overlay transparency to be freely changed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of a radar optoelectronic integrated display method based on GPU and CPU provided in an embodiment of the present application.

[0037] Figure 2 It is a flow chart of the optoelectronic video parsing and decoding method provided in an embodiment of the present application.

[0038] Figure 3 It is a schematic diagram of the structural composition of a radar optoelectronic integrated display system based on GPU and CPU provided in an embodiment of the present application.

[0039] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, the first radar video data and the second radar video data are used to distinguish different radar video data rather than to describe a specific order of the radar video data.

[0042] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0043] In the description of the embodiments of the present application, unless otherwise specified, "multi-channel" means two or more channels. For example, multi-channel radar video data refers to two or more radar video data, etc.

[0044] First, the technical terms involved in the embodiments of the present application are introduced.

[0045] API (Application Programming Interface), application programming interface.

[0046] OpenGL (Open Graphics Library) is a cross-language, cross-platform programming interface for rendering 2D and 3D vector graphics. It is a widely used standard in the field of computer graphics.

[0047] RTSP (Real-Time Streaming Protocol) is a network protocol used to control the transmission of audio and video streams. RTSP stream information refers to real-time streaming media data transmitted through the RTSP protocol.

[0048] The multimedia streaming library provides the necessary protocol support and encoding and decoding functions, and can handle a variety of audio and video encoding and decoding formats. It provides a framework that allows developers to create custom streaming client and server applications.

[0049] DMA (Direct Memory Access) is a hardware feature that allows input / output (I / O) devices to exchange data directly with a computer system's memory without involving the central processing unit (CPU).

[0050] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0051] like Figure 1 As shown, the radar optoelectronic integrated display method based on GPU and CPU provided in the embodiment of the present application specifically includes the following steps:

[0052] Step 1, read radar optoelectronic video data;

[0053] In this embodiment, the radar optoelectronic video data is acquired from the outside by the main processing CPU through the network, including multi-channel radar video data and multi-channel optoelectronic video data. The radar video data includes radar echo values, radar azimuth information and radar main pulses. The optoelectronic video data is compressed video in H264 or H265 format.

[0054] Step 2, generating a radar video hard layer and an optoelectronic video hard layer;

[0055] In this embodiment, according to the window size and position information written in the configuration files of radar video data and optoelectronic video data, the API provided by the autonomous GPU is called to generate the radar video hard layer and the optoelectronic video hard layer. In the above steps, if it is necessary to display all the multi-channel radar video data acquired at the same time, such as:

[0056] Radar video data 1, radar video data 2, ... radar video data n;

[0057] Photoelectric video data 1, photoelectric video data 2, ... photoelectric video data n.

[0058] The hard layer API is called accordingly to create radar video hard layer 1, radar video hard layer 2...radar video hard layer n, optoelectronic video hard layer 1, optoelectronic video hard layer 2...optoelectronic video hard layer n.

[0059] In this step, the multi-channel concurrent processing capability of the GPU is utilized to batch create hard layers according to the number of channels of received video data to achieve hardware acceleration.

[0060] In this embodiment, the radar video data needs to be preprocessed on the CPU side according to different data types, and the optoelectronic video needs to be analyzed.

[0061] Step 3, radar data preprocessing;

[0062] Radar video data preprocessing includes radar data decompression and data storage in radar data ring buffer.

[0063] Step 4, analysis of optoelectronic video data;

[0064] In this embodiment, the open source multimedia stream transmission library live555 is called on the CPU side to complete entropy decoding, reordering and other optical video stream parsing operations, such as Figure 2 As shown, the specific steps include:

[0065] Step 401, call the createNew function of the BasicTaskScheduler class member function to create a decoding task object scheduler.

[0066] Step 402, calling the BasicUsageEnrivonment class member function createNew to create an environment object env, wherein scheduler is passed in as a parameter of the createNew function.

[0067] Step 403, calling the RTSPClient class member function crateNew to create an RTSPClient instance, wherein the optoelectronic video URL address, the task object scheduler and the environment object env are passed in as parameters.

[0068] Step 404: call the RTSPClient class member function sendDescribeCommand to send an RTSP request.

[0069] Step 405, calling the DummySink class member function afterGettingFrame to receive a frame of optoelectronic video data, and storing the optoelectronic video data into the optoelectronic video buffer area.

[0070] Step 406, looping to receive rtsp stream data.

[0071] Steps 3 and 4 involve preprocessing and parsing of video data, which requires high computing power and can be better handled by the CPU.

[0072] Step 5, drawing radar images;

[0073] In this example, first, the GPU reads radar video data in batches into the GPU video memory through DMA; second, the OpenGL graphics drawing interface supported by the GPU is called to complete the drawing of the radar image.

[0074] Step 6, GPU decodes the optoelectronic video data;

[0075] In this example, first, the GPU reads the optoelectronic video data from the optoelectronic video buffer and calls the video decoding API function provided by the GPU to perform video decoding, such as Figure 2 As shown, decoding includes operations such as IQ inverse quantization (Inverse Quantization), IDCT inverse discrete cosine transform (Inverse Discrete Cosine Transform), intra-frame / inter-frame prediction and loop filtering.

[0076] Step 7, displaying the radar video hard layer and the photoelectric video hard layer;

[0077] In this example, the GPU calls the hard layer API interface to display the radar image on the corresponding radar video hard layer, and displays the decoded optoelectronic video data on the corresponding optoelectronic video hard layer.

[0078] Step 8, graphics window adjustment;

[0079] According to user needs, the size, position, overlay relationship and overlay transparency of the radar video hard layer window and the optoelectronic video hard layer window are adjusted. The window adjustment can be implemented by calling the API interface provided by the GPU.

[0080] In steps 5-8, batch tasks are processed by using the multi-channel concurrent processing capabilities of the GPU.

[0081] like Figure 3 As shown, the embodiment of the present application also provides a radar optoelectronic integrated display system based on GPU and CPU, which specifically includes the following parts:

[0082] A video data acquisition module, used to acquire radar video data and optoelectronic video data;

[0083] The hard layer generation module is used to call the GPU to create radar video hard layers and optoelectronic video hard layers;

[0084] The video data preprocessing module is used to call the CPU to preprocess the radar video data and analyze the optoelectronic video data;

[0085] A radar video display module is used to call the GPU to read the preprocessed radar video data, draw a radar image based on the radar video data, and display the radar image on the corresponding radar video hard layer through the hard layer API;

[0086] The optoelectronic video display module is used to call the GPU to read the parsed optoelectronic video data, and perform video decoding on the optoelectronic video data based on the video decoding API; and display the decoded optoelectronic video data on the corresponding optoelectronic video hard layer through the hard layer API;

[0087] The display adjustment module is used to call the GPU to adjust the windows of the radar video hard layer and the optoelectronic video hard layer.

[0088] It should be understood that the above-mentioned system is used to execute the methods in the above-mentioned embodiments. The implementation principles and technical effects of the corresponding modules in the system are similar to those described in the above-mentioned methods. The working process of the system can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0089] Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, such as Figure 4 As shown, the electronic device may include: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute the method in the above embodiment.

[0090] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0091] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0092] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0093] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0094] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0095] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium or transmitted through the storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0096] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0097] The above content is easily understood by those skilled in the art. The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A radar optoelectronic integrated display method based on GPU and CPU, characterized in that: include: S1. Acquire radar video data and optoelectronic video data; S2, calling GPU to create radar video hard layer and optoelectronic video hard layer; S3, calling the CPU to pre-process the radar video data and analyze the optoelectronic video data; S4, calling the GPU to read the preprocessed radar video data, and drawing a radar image based on the radar video data, and displaying the radar image on the corresponding radar video hard layer through the hard layer API; S5, calling the GPU to read the parsed optoelectronic video data, and performing video decoding on the optoelectronic video data based on the video decoding API; Display the decoded optoelectronic video data on the corresponding optoelectronic video hard layer through the hard layer API; S6. Calling the GPU to adjust the windows of the radar video hard layer and the optoelectronic video hard layer.

2. The radar optoelectronic integrated display method according to claim 1, characterized in that: In the step S1, the radar video data includes radar return data, radar azimuth data and radar main pulse data; the optoelectronic video data includes video data in various video compression formats.

3. The radar optoelectronic integrated display method according to claim 1, characterized in that: In step S2, a radar video hard layer or an optoelectronic video hard layer is created for each channel of radar video data or optoelectronic video data through the API of the GPU, the window size and position data are obtained from the configuration file in each channel of radar video data or optoelectronic video data, and the window size and position of the corresponding radar video hard layer or optoelectronic video hard layer are adjusted according to the window size and position data.

4. The radar optoelectronic integrated display method according to claim 1, characterized in that: In the step S3, the radar video data is preprocessed, specifically: calling the CPU to decompress the radar video data, and storing the decompressed radar video data into the radar data ring buffer.

5. The radar optoelectronic integrated display method according to claim 1, characterized in that: In the step S3, the optoelectronic video data is parsed, specifically: calling the CPU to read the optoelectronic video RTSP stream information from the configuration file of the optoelectronic video data, using the multimedia stream transmission library to perform parsing operations including entropy decoding and reordering on the optoelectronic video data, and storing the parsed optoelectronic video data in the optoelectronic data buffer.

6. The radar optoelectronic integrated display method according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S41, calling the GPU to read radar video data in batches from the radar data ring buffer into the GPU video memory through DMA; S42, using OpenGL to draw the radar video data in the video memory into a radar image; S43. Display the radar image on the corresponding radar video hard layer through the hard layer API.

7. The radar optoelectronic integrated display method according to claim 1, characterized in that: The step S5 specifically includes the following sub-steps: S51, calling the GPU to read the optoelectronic video data in batches from the optoelectronic data buffer into the GPU video memory; S52, performing decoding operations including IQ inverse quantization, IDCT discrete cosine inverse transform, intra-frame prediction, inter-frame prediction and loop filtering on the optoelectronic video data based on the video decoding API; S53. Display the decoded optoelectronic video data on the corresponding optoelectronic video hard layer through the hard layer API.

8. The radar optoelectronic integrated display method according to claim 1, characterized in that: The step S6 is specifically: adjusting the windows of the radar video hard layer and the optoelectronic video hard layer through the API of the GPU, and the adjustment includes: window movement, window scaling, window fusion and overlay, and changing window transparency.

9. A radar optoelectronic integrated display system based on GPU and CPU, characterized in that: include: A video data acquisition module, used to acquire radar video data and optoelectronic video data; The hard layer generation module is used to call the GPU to create radar video hard layers and optoelectronic video hard layers; The video data preprocessing module is used to call the CPU to preprocess the radar video data and analyze the optoelectronic video data; A radar video display module is used to call the GPU to read the preprocessed radar video data, draw a radar image based on the radar video data, and display the radar image on the corresponding radar video hard layer through the hard layer API; The optoelectronic video display module is used to call the GPU to read the parsed optoelectronic video data and perform video decoding on the optoelectronic video data based on the video decoding API; Display the decoded optoelectronic video data on the corresponding optoelectronic video hard layer through the hard layer API; The display adjustment module is used to call the GPU to adjust the windows of the radar video hard layer and the optoelectronic video hard layer.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the program stored in the memory. When the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1 to 8.

Images