Joint target detection system and detection method on water unmanned ship

Abstract

The application provides a kind of water surface unmanned ship joint target detection system and detection method, including photoelectric load, radar, embedded edge processing machine and central processing computer, wherein, photoelectric load, it is installed on unmanned ship;Radar, it is installed on unmanned ship;Embedded edge processing machine, with photoelectric load, radar, central processing computer are electrically connected, it is used to receive the video information obtained by photoelectric load and the radar information obtained by radar and carries out information fusion, and then sends the target information after fusion to central control computer;Central processing computer, with photoelectric load, radar and embedded edge processing machine are electrically connected, it is used to receive the video information of photoelectric load and the radar information of radar and carries out processing, or receives the information after fusion of embedded edge processing machine and carries out processing.The application can provide more accurate and detailed fusion information for the central control computer of water surface unmanned ship, system architecture upgrade is easy, and the application range is wider.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to a joint target detection system and method on a water surface unmanned ship. BACKGROUND

[0002] With the increasing frequent development and utilization of marine resources, the water surface unmanned ship has become an effective tool and an important development direction to support water operation. Radar, photoelectric load and sonar are the main sensors for perceiving the surrounding environment on the unmanned ship. Among them, the photoelectric load and radar equipment are usually installed on the upper part of the unmanned ship body, used for detecting the water surface targets near the ship body, and transmitting the detection results back to the respective data processing units. After the corresponding data processing is completed, the detection information of each sensor will be collected in the shipborne central control computer to form a unified water surface target distribution situation, which is used for the autonomous target tracking and navigation collision avoidance decision of the unmanned ship according to the task setting. Compared with land-based and air unmanned vehicles, the water surface unmanned ship is located on the sea, and the sea environment is complex and changeable. For example, the swell will cause a large swing of the video picture, the water mist will reduce the detection distance of the photoelectric sensor, and the sea clutter will interfere with the detection accuracy of the radar, affecting the target detection effect of the unmanned ship and the stable target tracking of the unmanned ship.

[0003] Compared with upgrading the individual software and hardware performance of radar and photoelectric, the method of improving the overall detection performance of the sensor group through coordination of the two for joint detection can effectively control the cost of the unmanned ship and increase the flexibility of the unmanned ship to adapt to different task needs. Among them, Wu Peng of the National University of Defense Technology designed a cooperative environmental perception method of unmanned ship navigation radar and photoelectric pod. The method obtains the final target distribution map by fusing the navigation radar target distribution map and the photoelectric pod target distribution map. The method aims to form a unified environmental information map of the detection information of the radar and the photoelectric pod. Since there is no deeper information fusion of the detected common targets, when the target information fed back by the sensors is inconsistent, it will cause target false alarm or missed detection. Zhu Xufang of the Naval University of Engineering proposed an information fusion unmanned ship detection and tracking method. The scheme matches and fuses the video image and the radar echo image by introducing spatial registration and time registration strategies. However, from the radar target features described in the scheme, which contain target height information, the method is not suitable for the two-coordinate navigation radar, but more suitable for the three-coordinate radar characteristics of large ships. SUMMARY

[0004] One purpose of the present application is to provide a joint target detection system on a water unmanned ship, which can more accurately obtain target comprehensive information, thereby enhancing the decision-making and autonomous control ability of the central control computer, and is more widely applicable, in view of the deficiencies of the prior art.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] A joint target detection system on a water unmanned ship, characterized in that it comprises an optical-electrical load, a radar, an embedded edge processing machine and a central processing computer, wherein,

[0007] The optical-electrical load is installed on the water unmanned ship and is used to obtain video images of the environment around the unmanned ship;

[0008] The radar is also installed on the water unmanned ship and is used to obtain radar signals of the environment around the unmanned ship;

[0009] The embedded edge processing machine is electrically connected with the optical-electrical load, the radar and the central processing computer, and is used to receive video information obtained by the optical-electrical load and radar information obtained by the radar and perform information fusion in the joint target detection mode, and then send the fused target information packet to the central processing computer for processing;

[0010] The central processing computer is electrically connected with the optical-electrical load, the radar and the embedded edge processing machine, and is used to receive video information obtained by the optical-electrical load and radar information obtained by the radar and perform processing in the conventional mode, and receive the fused information from the embedded edge processing machine and perform processing in the joint target detection mode.

[0011] Further, the embedded edge processing machine comprises a video decoding module, a radar data processing module, a video target detection module, an information fusion module and a data packaging module; wherein,

[0012] The video decoding module is used to decode the video signals transmitted by the optical-electrical load to obtain a single-frame image sequence that can be used for target detection;

[0013] The radar data processing module is used to complete the analysis and preprocessing of the front-end radar detection data packet, and package and send the processed radar detection data information and time information to the information fusion module;

[0014] The video target detection module is used to detect preset targets in the single-frame image sequence decoded by the video decoding module;

[0015] The information fusion module is used for fusing the radar data processed by the radar data processing module and the video target information detected by the video target detection module.

[0016] The data packaging module is used for packaging the target data and the detection time information fused by the information fusion module, and transmitting the data packet to the central control computer.

[0017] Further, the decoding mode of the video decoding module includes one or both of hardware decoding and software decoding.

[0018] Further, the fusion processing method of the information fusion module includes time registration, space registration and data fusion.

[0019] The time registration is used for calibrating the radar and video target data packets received by the information fusion module in time, so as to provide the radar and video target data information with consistent time stamps for subsequent space registration.

[0020] The space registration is used for completing the unified conversion of the target in the radar coordinate system, the camera coordinate system and the pixel coordinate system, and also performing corresponding coordinate conversion on the coordinates of the detected target according to the relative position relationship between the optoelectronic payload and the radar installation, so as to realize the unity of the detected target information in space.

[0021] The data fusion includes fusing the distance, azimuth and size information of the radar and video target which have completed time and space registration, so as to obtain more accurate and comprehensive target information than the former two.

[0022] Further, the optoelectronic payload is fixed on the mast of the water unmanned ship through the support table, the support table is vertically arranged with the mast, the radar is installed on the circular table at the top end of the mast of the water unmanned ship, and the projection points of the center points of the circular table at the top end of the mast and the support table on the deck plane of the unmanned ship coincide with or are parallel to the heading axis of the unmanned ship.

[0023] Another object of the present application is to provide a joint target detection method of the joint target detection system on the water unmanned ship.

[0024] Step 1: After receiving the switching mode instruction in the default regular working mode, the central processing computer sends the joint target detection working mode instruction to the embedded edge processor, the optoelectronic payload and the radar, and the embedded edge processor receives the data transmitted by the optoelectronic payload and the radar.

[0025] Step 2: The embedded edge processor decodes the video data of the optoelectronic payload to obtain a single frame image sequence, and detects the preset target in the decoded single frame image sequence.

[0026] Step 3: The embedded edge processor parses and preprocesses the received front-end radar detection data packets;

[0027] Step 4: The embedded edge processor adds the video target data information detected in Step 2 and the radar data information processed in Step 3 to two sets of buffer sequences, and performs registration according to the target data timestamp information in each sequence;

[0028] Step 5: The embedded edge processor performs spatial registration on the two sets of time-registered data, and then performs information fusion on the registered data in the pixel coordinate system;

[0029] Step 6: The embedded edge processor adds a registration timestamp to the fused target information, packages it, and sends it. The central processing computer receives the fused target information transmitted by the embedded edge processor, parses it, and displays it on the screen or uses it for other control decision operations as appropriate.

[0030] Furthermore, the method for the embedded edge processor to preprocess the front-end radar detection data packets in step 3 is as follows:

[0031] Step 3.1: Obtain the radar transmission. Each spoke data point contains [data points]. Each echo intensity data point is numbered as follows: The target dataset in the distance direction of the spoke data packet is In the formula, This represents the intensity value of the echo signal from the i-th target on the k-th spoke. The data are arranged sequentially in a two-dimensional pattern and plotted to obtain the radar detection map for that period. ;

[0032] Step 3.2: Apply Gaussian filtering to... Perform smoothing, and then... Binary processing is performed to obtain the radar detection binary image. ;

[0033] Step 3.3, for Label the connected regions on the graph, and let the labeled region be the first... The target connected regions are composed of In It consists of pixels, and the coordinate set is... ,in, Indicates the j-th target connected region. The x-coordinate of each pixel Indicates the j-th target connected region. The vertical coordinate of the pixel point, and the center of the connected region is set as the center of the target, then the distance between the target and the unmanned ship is:

[0034] ;

[0035] The orientation of the target is:

[0036] ;

[0037] The target spread is:

[0038] ;

[0039] Wherein, is the detection distance, and are the maximum and minimum values of the horizontal coordinate in .

[0040] Further, the method of registration according to the time stamp in step 4 is:

[0041] The Lagrange three-point difference method is used to interpolate and pair the target data packets from the radar and the photoelectric load, and the specific pairing calculation formula is as follows:

[0042] ;

[0043] Wherein, is the time to be interpolated, , , is the data time stamp participating in interpolation, is the radar or photoelectric load target measurement data value of the point to be interpolated, , , correspond to the target measurement data at , , .

[0044] Further, the spatial registration method in step 5 is:

[0045] The pinhole imaging model of the local coordinate system to the photoelectric load pixel coordinate system is used to transform the radar target data coordinates to the pixel coordinate system where the photoelectric load is located:

[0046] = ·

[0047] Wherein, The coordinates of the radar target center in the local spatial coordinate system are ( , () represents the coordinates of the projected radar target center in the pixel coordinate system. , = , , The internal parameters of the built-in camera in the photoelectric payload. The focal length of the camera. and These are the actual physical dimensions of the camera's imaging unit.

[0048] Furthermore, the information fusion method in step 5 is as follows:

[0049] Assume the pixel coordinates of the center point of one of the radar targets are ( , ), then its ROI in the pixel coordinate system is represented by the coordinate point ( , Centered on ) and expanding Expand from the original target The formula for calculating the broadening of the photoelectric payload target, obtained through pinhole imaging model transformation, is as follows:

[0050] ;

[0051] in, and Here, represents the pixel coordinates of the top-left and bottom-right anchor points of the annotation box, respectively. In the pixel coordinate system, the IOU calculation formula for the target broadening of the radar and electro-optical payload is as follows:

[0052] ;

[0053] Set threshold ,when When the overlapping area of ​​the target detected by the radar and the optoelectronic payload meets the conditions, the result after data fusion is accurate, thus expanding the communication between optoelectronic and radar. Category, confidence level information, and radar azimuth and range information are combined as the final fused data; when If the overlapping area of ​​the target detected by the radar and the optoelectronic payload does not meet the conditions, the result after data fusion may not be accurate. Therefore, the following judgment should be made: If... , This indicates that the photoelectric payload did not detect the target, and the radar data will be used as the final fused data; if , This indicates that the radar did not detect the target, and the photoelectric data is used as the final fused data; if , Then, the broadening and integration of photoelectric and radar technologies... , class, confidence information, central direction of the azimuth of the radar and the electro-optical payload , distance information together as final fusion data; wherein, The calculation formula is as follows:

[0054] ;

[0055] In the formula, The azimuth of the target detected by the radar is represented by The azimuth of the target detected by the electro-optical payload is represented by The smaller one of the two azimuth angles is represented by

[0056] Compared with the prior art, the beneficial effects of the present application are:

[0057] The present application externally connects an embedded edge processing machine (including a video decoding module, a radar data processing module, a video target detection module, an information fusion module and a data packaging module) between the central control computer of the unmanned ship and the radar and the electro-optical payload, detects preset targets in the video images and spoke data output by the electro-optical payload and the radar quickly, and performs time and space registration on the coordinates, azimuths, distances and the like of the two groups of target data, finally performs information fusion processing on the registered data to obtain more accurate and detailed fusion data to support the central control computer to make decision and judge and to perform autonomous control.

[0058] The advantages of the present application are: first, the embedded edge processing machine has more professional video processing hardware, compared with the general central control computer, can more real-time and efficient complete target detection, second, can maintain the integrity of the original system of the unmanned ship to the greatest extent, since the radar and video data signals are transmitted into the embedded edge processing machine in a branching manner, do not affect the hardware architecture and software process of the original system, third, the invention is designed for commonly used unmanned ship navigation radar and single-channel video electro-optical payload, has better application promotion. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is the installation position diagram of the navigation radar and the electro-optical payload of the water surface unmanned vehicle of the embodiment of the present application, wherein (a) is an isometric view and (b) is a front view;

[0060] Figure 2 is the principle block diagram of the joint target detection system of the unmanned vehicle of the embodiment of the present application;

[0061] Figure 3 is the hardware connection diagram of the joint target detection system of the unmanned vehicle of the embodiment of the present application;

[0062] Figure 4A joint target detection system mode information interaction diagram of an embodiment of the present application;

[0063] Figure 5 A joint target detection information fusion flowchart of an unmanned vehicle of an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0065] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0066] The present application will be further described in conjunction with specific embodiments, but is not limited by the embodiments.

[0067] The navigation radar and the photoelectric load on a typical water surface unmanned ship are usually used independently, and two-way target data are generally processed and displayed separately. Since the radar and the photoelectric load have advantages and disadvantages in target detection, the navigation radar has weak detection capability for small water surface targets and floating objects, cannot obtain target height information, and is easily disturbed by clutter and noise when close to land; the photoelectric load is easily affected by light, water mist and weather, and has relatively poor detection distance and angle resolution. Although the "man-in-the-loop" remote control method can meet part of the task requirements, it weakens the characteristics of autonomous control of the unmanned ship. When the unmanned ship is far away from the command platform, the communication bandwidth and communication time delay will greatly affect the effect of the "man-in-the-loop" method, and reduce the flexibility and reliability of the unmanned ship when autonomously performing tasks.

[0068] Generally, the scheme of upgrading the original equipment with high-performance radar and electro-optical payload can also enhance the perception range of the unmanned ship to the surrounding environment, increase its ability to resist special environment, but the cost is high, the cost-effectiveness is low, and it is not suitable for small low-configured and cost-limited unmanned ships. Therefore, using multi-information fusion means is an effective way to effectively improve the overall perception and autonomous decision-making performance of the unmanned ship. The similar invention scheme that can be consulted is mostly based on the unmanned ship equipped with laser radar, but the laser radar has short detection distance and is easily affected by weather, and is mostly suitable for collision avoidance and automatic berthing in still water and narrow channel; the electro-optical payload in the scheme of the National University of Defense Technology needs to have laser ranging function, and the radar in the scheme of the Naval University of Engineering needs to have three-coordinate target detection capability, which limits its use range to a certain extent. Therefore, the embodiment of the present application is designed based on the commonly used two-dimensional navigation radar and typical electro-optical payload as equipment, aiming to obtain a joint target detection system scheme with better engineering universality and feasibility. Specifically, the embodiment of the present application provides a joint target detection system on an unmanned vehicle, which comprises an electro-optical payload, a radar, an embedded edge processor and a central processing computer. In the embodiment, the electro-optical payload is installed on the mast of the unmanned ship, specifically, the electro-optical payload is fixed on the mast of the unmanned ship through a support table, and the support table is vertically arranged with the mast, and the electro-optical payload is used to obtain video images of the environment around the unmanned vehicle. The radar is installed on the circular table at the top end of the mast of the unmanned ship, and is used to obtain radar signals of the environment around the unmanned ship. The radar can detect the 360-degree environmental targets around the ship body. In order to improve the fusion effect of the radar and the video data and reduce the calculation error caused by unreasonable installation position in data space registration, the circular table at the top end of the mast, the plane of the support table and the deck plane of the unmanned ship are physically leveled, and the projection points of the center points of the circular table at the top end of the mast and the support table on the deck plane of the ship are connected with the ship's heading axis, so as to ensure the consistency of the zero point direction of the navigation radar and the electro-optical payload, as shown in Figure 1

[0069] The embedded edge processor is electrically connected with the electro-optical payload, the radar and the central processing computer, and is used to receive the video information obtained by the electro-optical payload and the radar information obtained by the radar for information fusion in the joint target detection working mode, and send the fused target information packet to the central control computer for processing. The central processing computer is electrically connected with the electro-optical payload, the radar and the embedded edge processor, and is used to receive and process the video information obtained by the electro-optical payload and the radar information obtained by the radar in the conventional working mode, and receive and process the fused information of the embedded edge processor in the joint target detection working mode.

[0070] Figure 2 The principle block diagram of the joint target detection system of the unmanned vehicle of the embodiment is shown. Figure 2 ​In the middle, the left dashed box is the original system equipment of the unmanned ship, and the right solid box is the embedded edge processor externally connected. For the sake of simplicity, the photoelectric load in the following text refers to a single-channel video photoelectric load unless otherwise specified, and the processor refers to the embedded edge processor. In the left block diagram, the photoelectric load and the radar output two identical signals respectively, wherein data channel 1-1 and data channel 1-2 are original data channels of the unmanned ship system, which directly transmit video signals and radar signals to the shipborne central control computer for processing. Data channel 2-1 and data channel 2-2 are newly added data channels under the condition of joint target detection, which transmit video signals and radar signals to the video decoding module and the radar data processing module in the externally connected processor for processing. By Figure 1 It can be seen that the processor can work independently at the software and hardware level, and will not affect the structure and working process of the original system, and has good modularization characteristics.

[0071] The embedded edge processor comprises a video decoding module, a radar data processing module, a video target detection module, an information fusion module and a data packaging module. The video decoding module can adopt hardware decoding or software decoding mode and has the capability of decoding mainstream video signals such as H.263, H.264 and H.265 / HEVC, wherein the hardware decoding has higher processing speed and can improve the overall response speed of the system to the target. The video decoding module is mainly responsible for decoding the video signal transmitted by the photoelectric load in the processor to obtain a single-frame image sequence that can be used for target detection. The video target detection module is used for detecting preset water surface targets in the decoded image sequence, and the selection of the detection algorithm can be determined by the specific hardware configuration and task requirements, including traditional inter-frame difference method, background difference method, ViBe algorithm and other algorithms that distinguish features such as grayscale, brightness, color and shape, or deep learning-based target detection algorithms such as SSD series or YOLO series target detection algorithms, and the type of target detected is determined according to the specific work task. The detected target position, contour anchor point, pixel size and other information are sent to the information fusion module together with the time stamp information after packaging. The radar data processing module mainly completes the analysis and preprocessing of the front-end radar detection data packet, which mainly consists of a radar image filtering program and a target parameter calculation process, wherein the image filtering program mainly completes the radar data image smoothing processing, and typical linear smoothing filters such as Gaussian filter, mean filter and median filter can be selected, and the target parameter calculation completes the determination of the data connected region and its corresponding distance and azimuth. The processed radar detection data information will be packaged together with the time information and sent to the information fusion module. The information fusion module is mainly responsible for completing the fusion processing of radar data and photoelectric target information, including three steps of time registration, space registration and data fusion. Among them, the time registration mainly calibrates the radar and video target data packets received by the information fusion module in time, provides radar and video target data information with consistent time stamps for subsequent space registration, and the time registration algorithm that can be used includes commonly used Taylor expansion correction method, interpolation and extrapolation method and virtual fusion method; the space registration mainly completes the unified conversion of the target in the radar coordinate system, the camera coordinate system and the pixel coordinate system, and the typical camera pinhole model can be used for calculation, and at the same time, according to the relative position relationship between the photoelectric load and the radar installation, the coordinates of the detected target are converted, and the space unified of the detected target information of the two is realized. Data fusion is to fuse the radar and video target distance, azimuth and size information that have completed time and space registration to obtain more accurate and comprehensive target information than the former two, and the data fusion can adopt least square method, similarity correlation method and Kalman filter method according to specific application requirements.The data packaging module is responsible for packaging the fused target data and the checkout time information, and transmitting the data package to the unmanned ship central control computer. In the embodiment, see. Figure 3 The hardware sources are as follows: a JHP-103 photoelectric load produced by the company Giant, a Halo20 navigation radar of the company SIMRAD, a Nport 5232 serial server of the company MOXA, an H.264 / H.265 video codec processor, and a control module based on Jetson xavier and NANO of the company Nvidia, and an EPC-B embedded industrial computer of the company Advantech. Among them, the photoelectric load sends the working state data to the serial server in a timely manner and responds to the control instructions sent by the serial server. The radar sends the working state and detection data to the control module in a timely manner and responds to the control instructions sent by the control module. The video codec encodes the video signal sent by the photoelectric load and sends it to the network router. The processor receives the data information from the photoelectric load and the radar, fuses the information, and sends the new data package to the network router. Meanwhile, the processor also receives the control instructions from the network router sent by the central processing computer. The central processing computer obtains the state and data information from the photoelectric load, the radar, and the processor through the network router, and sends the control instructions according to the task requirements.

[0072] The joint target detection system of the embodiment includes two working modes after the processor is added, Figure 4 The working mode information interaction diagram of the joint target detection system is shown, the first one is the regular working mode, mainly used in unmanned ship navigation, regular patrol and other tasks, and the second one is the joint target detection working mode, mainly used in reconnaissance and evidence collection, target tracking and other tasks. The biggest difference between the two working modes is that the processor fuses the incoming data of the photoelectric load and the radar and transmits the target fusion data package to the central control computer. From Figure 4 It can be seen that the unmanned ship can be switched between the two working modes at will, and the addition of the processor is only a modular upgrade of the original system function, and does not affect the integrity of the original system independent function.

[0073] The regular working mode (default): after the power of the unmanned ship system is turned on, the central control computer, the photoelectric load, the radar and the processor are started, and after the self-checking is completed, they enter the standby mode. After receiving the working instruction, the central control computer sends the working mode instruction to the photoelectric load, the radar and the processor. After receiving the instruction, the photoelectric load and the radar are converted to the regular working mode, and then the obtained image data and radar data are transmitted back to the central control computer. The central control computer processes and displays the returned data by using the installed software program, and the processor continues to remain in the standby state after receiving the working mode instruction.

[0074] Joint target detection working mode: the unmanned ship working in the normal working mode receives new working instructions, and simultaneously issues working mode instructions to the photoelectric load, radar and processor. After receiving the instructions, the photoelectric load and the radar switch to the joint target detection working mode, and copy the obtained video images and radar data. The two-way data is sent to the central control computer and the processor respectively. After receiving the instructions, the processor enters the working state from the standby state, and performs information fusion on the two-way data transmitted, and sends the fused target fusion data packet to the central control computer for processing. The central control computer processes the original video, radar and fused data at the same time, and displays the data on the display panel according to the specific requirements.

[0075] The embodiment of the application also provides a joint target detection method of a joint target detection system on a water surface unmanned ship, as shown in the figure, comprising the following steps: Figure 5

[0076] Step 1: the central processing computer receives the remote / local switching working mode instruction in the default normal working mode, and immediately issues the joint target detection working mode instruction to the processor, photoelectric load and radar. The processor starts to receive the data transmitted by the photoelectric load and the radar through the local network.

[0077] Step 2: the photoelectric load video data in the application is encoded in H.265 format, and after entering the processor through the local network of the unmanned ship, the video decoding is first completed by the built-in hardware decoding module of Jetson xavier. The processed video data enters the target detection module in the form of single frame image for preset target detection. Since Jetson xavier has strong image processing performance, the embodiment adopts YOLO v8 version with good real-time performance and detection performance as the core algorithm of the target detection module (the YOLO algorithm can be upgraded in the future to further improve the target detection performance). The target type of the target detection weight model is multiple ships determined according to the task. The first detection target result can be expressed as:

[0078] (1)

[0079] wherein, is the pixel coordinate of the upper left anchor point of the annotation box, is the coordinate of the lower right corner anchor point of the annotation box, is the type number of the target, is the confidence of target detection, and the azimuth angle of the target The calculation formula is:

[0080] (2) ​​

[0081] in The heading angle of the unmanned vessel in the local coordinate system. The pointing angle of the optical axis of the photoelectric load (clockwise from zero).

[0082] Step 3: The radar data processing module and the target detection module work in parallel. In this embodiment, the Halo 20 radar echoes are sent to the local area network via UDP multicast. The radar antenna, with the bow pointing as the starting point, is scanned 360 degrees clockwise to obtain a set of numbers from 0 to... SpokeData package ( (The radar azimuth resolution is set by the user). By stitching the above data together in sequence, the radar image of this scan can be obtained. (Based on azimuth resolution...) Distance resolution Maximum detection range m, noise suppression strength The calculation is illustrated using an example, with the number being [number]. The azimuth angle represented by the spokes in the local coordinate system The calculation formula is:

[0083] (3)

[0084] in, The heading angle of the unmanned vessel. The spoke number is indicated by `mod`, which represents the modulo operation on the two numbers before and after the parentheses. Each spoke data item contains... Echo intensity data (intensity) ), then the number is The target dataset in the distance direction of the spoke data packet is In the formula, This represents the intensity value of the echo signal from the i-th target on the k-th spoke. By arranging the data sequentially in a two-dimensional pattern and plotting it, a radar detection map for that period can be obtained. .

[0085] Step 4: In this embodiment, a Gaussian filter is first used to... Smoothing is performed, and then dilation and erosion methods from graphic morphology are used to smooth the surface. To reduce computational load, noise suppression is performed. Binary processing is performed to obtain the radar detection binary image. .by For example, the target distribution dataset in the distance direction is as follows: , For the first The first of the spokes 1 and 0, respectively:

[0086] (4)

[0087] Step 5: Label the connected regions on the , and let the th target connected region labeled by consists of pixel points, and the coordinate set is , where is the horizontal coordinate of the th pixel point in the th target connected region, is the vertical coordinate of the th pixel point in the

[0088] th target connected region, and let the centroid of the connected region be the center of the target, then the distance between the target and the unmanned ship is: (5)

[0089] The orientation of the target is:

[0090] (6)

[0091] The target spread is:

[0092] (7)

[0093] where is the detection distance, and are the maximum and minimum values of the horizontal coordinate in .

[0094] Step 6: The information fusion module adds the target data information from the target detection module and the radar data processing module to the two groups of cache sequences, and first aligns according to the target data time stamp information in each sequence. In this embodiment, Lagrange three-point difference method is used to interpolate and pair the target data packets from radar and photoelectric load, and the specific pairing calculation formula is as follows:

[0095] (8)

[0096] where is the time to be interpolated, , , are the data time stamps involved in interpolation, is the radar or photoelectric load target measurement data value of the point to be interpolated. 、 、 correspond to the target measurement data at the time of 、 、 .

[0097] Step 7: Spatially register the two sets of data after time registration. The pinhole imaging model of the local coordinate system to the pixel coordinate system of the electro-optical payload is used to transform the radar target data coordinates to the pixel coordinate system of the electro-optical payload:

[0098] = · (9)

[0099] wherein, is the coordinate value of the radar target center in the local spatial coordinate system, (xR, yR) is the coordinate value of the projected radar target center in the pixel coordinate system, , , , = , , is the internal parameter of the built-in camera of the electro-optical payload, is the focal length of the camera, and are the real physical dimensions of the camera imaging unit, respectively.

[0100] Step 8: Information fusion of the registered data in the pixel coordinate system. Assuming that the pixel coordinates of one of the radar target center points are (xR, yR), the ROI in the pixel coordinate system is centered at the coordinate point (xR, yR) and is expanded by , , , , , , ,

[0101] ; (10)

[0102] wherein, and are the pixel coordinates of the top-left and bottom-right anchor points of the bounding box, respectively. In the pixel coordinate system, the IOU calculation formula of the radar and electro-optical payload target width is:

[0103] ; (11)

[0104] Set the threshold value , When the overlapping area of ​​the target detected by the radar and the optoelectronic payload meets the conditions, the result after data fusion is accurate, thus expanding the communication between optoelectronic and radar. The category, confidence level information, and radar azimuth and range information are combined as the final fused data. If the overlapping area of ​​the target detected by the radar and the optoelectronic payload does not meet the conditions, the result after data fusion may not be accurate. Therefore, the following judgment is made: If... , This indicates that the photoelectric payload did not detect the target, and the radar data will be used as the final fused data; if , This indicates that the radar did not detect the target, and the photoelectric data is used as the final fused data; if , Then, the broadened union of the optoelectronic payload and the radar is used. Category, confidence level information, and center direction of the radar and optoelectronic payload. Distance information, along with other data, is used as the final fused data. The calculation formula is as follows: This indicates taking the smaller of the two azimuth angles:

[0105] (12);

[0106] Set threshold ,when When the overlapping area of ​​the target detected by the radar and the optoelectronic payload meets the conditions, the result after data fusion is accurate, thus expanding the communication between optoelectronic and radar. Category, confidence level information, and radar azimuth and range information are combined as the final fused data; when If the overlapping area of ​​the target detected by the radar and the optoelectronic payload does not meet the conditions, the result after data fusion may not be accurate. Therefore, the following judgment should be made: If... , This indicates that the photoelectric payload did not detect the target, and the radar data will be used as the final fused data; if , This indicates that the radar did not detect the target, and the photoelectric data is used as the final fused data; if , Then, the broadening and integration of photoelectric and radar technologies... Category, confidence level information, and center direction of the radar and electro-optical payload. Distance information, together with other data, is used as the final fused data; among them, The calculation formula is as follows:

[0107] ;

[0108] In the formula, represents the azimuth of the target detected by the radar, represents the azimuth of the target detected by the photoelectric load, represents the azimuth of the target detected by the photoelectric load,

[0109] Step 9: The fusion data packaging module packages the fused target azimuth, distance, spread, and category information together with the registered time stamp, and sends it to other designated devices by the data sending process. The central processing computer analyzes the received fusion target information and displays it on the screen or uses it for other control decision operations according to the situation.

[0110] The above is only the preferred embodiment of the present application, and does not limit the implementation and protection scope of the present application. For those skilled in the art, it should be realized that any equivalent replacement and obvious changes made by applying the content of the present application should be included in the protection scope of the present application.

Claims

1. A joint target detection method for a joint target detection system on an unmanned surface vessel, characterized in that, This includes optoelectronic payloads, radar, embedded edge processors, and central processing computers; among which, The optoelectronic payload is installed on an unmanned surface vessel to acquire video images of the environment surrounding the unmanned vessel. Radar, which is also installed on unmanned surface vessels, is used to acquire radar signals of the environment around the unmanned vessels. The embedded edge processor is electrically connected to the optoelectronic payload, radar, and central processing computer. In the joint target detection mode, it receives video information obtained by the optoelectronic payload and radar information obtained by the radar, performs information fusion, and then sends the fused target information packet to the central processing computer for processing. The central processing computer is electrically connected to the optoelectronic payload, radar, and embedded edge processor. In normal working mode, it receives and processes video information obtained by the optoelectronic payload and radar information obtained by the radar. In joint target detection working mode, it receives and processes the information fused by the embedded edge processor. The joint target detection method includes the following steps: Step 1: In the default normal working mode, after receiving the working mode switching instruction, the central processing computer sends the instruction to the embedded edge processor, optoelectronic payload and radar to enter the joint target detection working mode. The embedded edge processor receives the data transmitted from the optoelectronic payload and radar. Step 2: The embedded edge processor decodes the received photoelectric payload video data to obtain a single-frame image sequence, and detects the preset target in the decoded single-frame image sequence; Step 3: The embedded edge processor parses and preprocesses the received front-end radar detection data packets; Step 4: The embedded edge processor adds the video target data information detected in Step 2 and the radar data information processed in Step 3 to two sets of buffer sequences, and performs registration according to the target data timestamp information in each sequence; Step 5: The embedded edge processor performs spatial registration on the two sets of time-registered data, and then performs information fusion on the registered data in the pixel coordinate system; Step 6: The embedded edge processor adds a registration timestamp to the fused target information, packages it, and sends it. The central processing computer receives the fused target information transmitted from the embedded edge processor, parses it, and displays it on the screen or uses it for other control decision operations as appropriate. The method for preprocessing the front-end radar detection data packets by the embedded edge processor in step 3 is as follows: Step 3.1: Obtain the radar transmission. Each spoke data point contains [data points]. Each echo intensity data point is numbered as follows: The target dataset in the distance direction of the spoke data packet is In the formula, This represents the intensity value of the echo signal from the i-th target on the k-th spoke. The data are arranged sequentially in a two-dimensional pattern and plotted to obtain the radar detection map within the period. ; Step 3.2: Apply Gaussian filtering to... Perform smoothing, and then... Binary processing is performed to obtain the radar detection binary image. ; Step 3.3, for Label the connected regions on the graph, and let the labeled region be the first... The target connected regions are composed of In It consists of pixels, and the coordinate set is... ,in, Indicates the j-th target connected region. The x-coordinate of each pixel Indicates the j-th target connected region. Given the y-coordinates of 1 pixel, and assuming the centroid of the connected region is the center of the target, then its distance from the unmanned vessel is: ； The target's location is: ； The target is broadened to: ； in, To detect distance, and They are respectively middle x-axis The maximum and minimum values; The information fusion method in step 5 is as follows: Assume the pixel coordinates of the center point of one of the radar targets are ( , ), then its ROI in the pixel coordinate system is represented by the coordinate point ( , Centered on ) and expanding Expand from the original target The formula for calculating the broadening of the photoelectric payload target, obtained through pinhole imaging model transformation, is as follows: ； in, and Here, represents the pixel coordinates of the top-left and bottom-right anchor points of the annotation box, respectively. In the pixel coordinate system, the IOU calculation formula for the target broadening of the radar and electro-optical payload is as follows: ； Set threshold ,when When the overlapping area of ​​the target detected by the radar and the optoelectronic payload meets the conditions, the result after data fusion is accurate, thus expanding the communication between optoelectronic and radar. Category, confidence level information, and radar azimuth and range information are combined as the final fused data; when If the overlapping area of ​​the target detected by the radar and the optoelectronic payload does not meet the conditions, the result after data fusion may not be accurate. Therefore, the following judgment should be made: If... , This indicates that the photoelectric payload did not detect the target, and the radar data will be used as the final fused data; if , This indicates that the radar did not detect the target, and the photoelectric data is used as the final fused data; if , Then, the broadening and integration of photoelectric and radar technologies... Category, confidence level information, and center direction of the radar and optoelectronic payload. Distance information, together with other data, is used as the final fused data; among them, The calculation formula is as follows: ； In the formula, This indicates that the radar has detected the target's location. This indicates that the photoelectric payload has detected the target's location. This indicates that the smaller of the two azimuth angles is taken.

2. The joint target detection method of the joint target detection system on an unmanned surface vessel according to claim 1, characterized in that, The embedded edge processor includes a video decoding module, a radar data processing module, a video target detection module, an information fusion module, and a data packaging module; wherein, The video decoding module is used to decode the video signal transmitted from the photoelectric payload to obtain a single-frame image sequence that can be used for target detection; The radar data processing module is used to parse and preprocess the front-end radar detection data packets, and then package the processed radar detection data information and time information and send them to the information fusion module. The video target detection module is used to detect preset targets in a single-frame image sequence decoded by the video decoding module; The information fusion module is used to fuse the radar data processed by the radar data processing module with the video target information detected by the video target detection module. The data packaging module is used to package the target data and detection time information fused by the information fusion module and transmit the data packet to the central control computer.

3. The joint target detection method of the joint target detection system on an unmanned surface vessel according to claim 2, characterized in that, The video decoding module can use one or both of the following decoding methods: hardware decoding and software decoding.

4. The joint target detection method of the joint target detection system on an unmanned surface vessel according to claim 2, characterized in that, The fusion processing methods of the information fusion module include time registration, spatial registration, and data fusion; Among them, time registration is to calibrate the radar and video target data packets received by the information fusion module in time, so as to provide radar and video target data information with consistent timestamps for subsequent spatial registration; Spatial registration is to complete the unified transformation of the target in the radar coordinate system, camera coordinate system and pixel coordinate system. At the same time, it is also necessary to perform corresponding coordinate transformation on the coordinates of the detected target according to the relative position relationship between the photoelectric payload and the radar installation, so as to achieve the spatial unification of the detected target information of the two. Data fusion involves fusing information on the distance, azimuth, and size of radar and video targets that have been registered in time and space to obtain more accurate and comprehensive target information than the former two.

5. The joint target detection method of the joint target detection system on an unmanned surface vessel according to claim 1, characterized in that, The optoelectronic payload is fixed to the mast of the unmanned surface vessel via a support platform, which is perpendicular to the mast. The radar is installed on a circular platform at the top of the mast, and the line connecting the center point of the circular platform at the top of the mast and the center point of the support platform on the deck plane of the unmanned surface vessel coincides with or is parallel to the heading axis of the unmanned surface vessel.

6. The joint target detection method of the joint target detection system on an unmanned surface vessel according to claim 1, characterized in that, The method for registration based on timestamps in step 4 is as follows: The Lagrange three-point interpolation method is used to interpolate and pair target data packets from radar and electro-optical payloads. The specific pairing calculation formula is as follows: ； in, For the time to be interpolated, , , For the timestamps of the data used in the interpolation, The target measurement data value of the radar or optoelectronic payload at the point to be interpolated. , , They correspond to as , , Target measurement data at any given time.

7. The joint target detection method of the joint target detection system on an unmanned surface vessel according to claim 1, characterized in that, The spatial registration method in step 5 is as follows: The pinhole imaging model, which transforms the radar target data coordinates to the pixel coordinates of the optoelectronic payload using the local coordinate system, is used to transform the radar target data coordinates to the pixel coordinates of the optoelectronic payload. = · in, The coordinates of the radar target center in the local spatial coordinate system are ( , () represents the coordinates of the projected radar target center in the pixel coordinate system. , = , , The internal parameters of the built-in camera in the photoelectric payload. The focal length of the camera. and These are the actual physical dimensions of the camera's imaging unit.

Images