Distributed three-light real-time panoramic splicing device and splicing method

Abstract

The invention discloses a distributed three-light real-time panoramic splicing device and a splicing method. A splicing unit and four same three-light fusion sensing units which are arranged in a distributed mode are arranged; the three-light fusion sensing unit comprises an infrared thermal imager, a visible light camera, a laser radar and a fusion module; the fusion module carries out fusion processing on an image acquired by the thermal infrared imager, an image acquired by the visible light camera and a point cloud image acquired by the laser radar, and sends three-light fusion image data after fusion processing to the splicing unit through the Ethernet in a UDP protocol mode; and the splicing unit is used for splicing the three-light fusion images sent by the four-path fusion module to obtain a panoramic image. The device realizes panoramic display of the scene through fusion of the three-light data, further realizes day and night monitoring of the scene, and improves the accuracy of panoramic image target display.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion and splicing, and in particular relates to a distributed three-light real-time splicing device and a splicing method. Background Art

[0002] With the development of optoelectronic technology, visible light cameras and lidar are widely used in vehicle-mounted smart devices. However, visible light cameras and lidar cannot meet the needs of day and night monitoring without lighting conditions. Most existing panoramic imaging systems are based on 360-degree panoramic scanning, with a small vertical field of view and high output delay. They cannot complete panoramic display and real-time full-field day and night monitoring tasks, and cannot meet the task requirements of specific scenarios.

[0003] A new real-time splicing device is needed to solve the above technical problems. Summary of the invention

[0004] The purpose of the present invention is to provide a distributed three-light real-time splicing device to solve the technical problems in the prior art that the accuracy of panoramic images is not high and day and night monitoring cannot be achieved.

[0005] The present invention also aims to provide a distributed three-light real-time splicing method.

[0006] The technical solution of the present invention to solve the technical problem is: A distributed three-light real-time stitching device comprises: a stitching unit, four identical three-light fusion sensing units which are distributed; the three-light fusion sensing unit comprises an infrared thermal imager, a visible light camera, a laser radar, and a fusion module; the fusion module performs fusion processing on an image collected by the infrared thermal imager, an image collected by the visible light camera, and a point cloud image collected by the laser radar, and sends the fused three-light fusion image data to the stitching unit via Ethernet; the stitching unit stitches the three-light fusion images sent by the four-way fusion modules to obtain a panoramic image.

[0007] Preferably, the fusion module includes: The first processing module pre-processes the images collected by the infrared thermal imager and the images collected by the visible light camera, and sends the pre-processed image data to the fusion processing module and the multi-modal recognition module respectively; The second processing module pre-processes the point cloud image collected by the laser radar; sends the pre-processed image data to the fusion processing module and the multimodal recognition module respectively; superimposes the three-light image target frame on the three-light fusion image, and sends the superimposed three-light fusion image to the fusion processing module; and sends the encoded three-light fusion image to the splicing unit via Ethernet using the UDP protocol; The fusion processing module fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the laser radar, and sends the obtained three-light fusion image to the second processing module; according to the sequence of the image frame IDs formed by the three-light fusion image, they are sequentially put into a queue, and sequentially encoded according to the queue sequence to obtain the encoded three-light fusion image; and the encoded three-light fusion image is sent to the second processing module; The multimodal recognition module performs multimodal recognition processing on the image data collected by the infrared thermal imager, the image data collected by the visible light camera, and the point cloud data collected by the laser radar, determines whether the three-light images meet the requirements, and sends the three-light image target frames that meet the requirements to the second processing module; The output end of the laser radar is connected to the input end of the second processing module, the output end of the second processing module is connected to the input end of the splicing unit, the second processing module is connected to the fusion processing module and the multimodal recognition module, the fusion processing module and the multimodal recognition module are interconnected and respectively connected to the first processing module; the first processing module is connected to the infrared thermal imager, the visible light camera, and the splicing unit.

[0008] Preferably, the splicing unit comprises: The receiving module decodes and receives the encoded three-light fusion image and sends it to the processing module; The processing module sequentially performs numbering processing, image edge smoothing and filtering processing, and splicing processing on the received four-way three-light fusion images to obtain a panoramic image. Preferably, the splicing process adopts a texture splicing method.

[0009] Preferably, the infrared thermal imager, the visible light camera and the first processing module transmit image data via a communication interface; and the laser radar and the second processing module transmit point cloud data via Ethernet.

[0010] Preferably, the splicing unit sends a synchronization signal to four fusion modules, and the four fusion modules send the synchronization signal to the infrared thermal imager, the visible light camera, and the lidar respectively.

[0011] Preferably, it also includes: a display unit for displaying the panoramic image; including three modes: single-frame display, 180° up and down, and panoramic display.

[0012] A distributed three-light real-time splicing method comprises the following steps: S1: The fusion module in the three-light fusion perception unit fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the lidar, and sends the fused three-light fusion image data to the splicing unit via Ethernet; S2: The stitching unit sequentially performs numbering processing, image edge smoothing and filtering processing, and stitching processing on the three-light fusion images sent by the four-way fusion module by means of texture stitching to obtain a panoramic image.

[0013] Preferably, the step S1 specifically comprises: S1.1: The first processing module performs image smoothing, filtering, and distortion correction processing on the images collected by the infrared thermal imager and the visible light camera; the second processing module performs point cloud filtering and distortion correction processing on the point cloud image collected by the laser radar; the first processing module and the second processing module send the processed image data to the fusion processing module and the multimodal recognition module respectively; S1.2: The fusion processing module fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the lidar, and sends the obtained three-light fusion image to the second processing module; The multimodal recognition module performs multimodal recognition processing on the image data collected by the infrared thermal imager, the image data collected by the visible light camera, and the point cloud data collected by the laser radar, determines whether the three-light images meet the requirements, and sends the three-light image target frames that meet the requirements to the second processing module; S1.3: The second processing module superimposes the three-light image target frame on the three-light fusion image, and sends the superimposed three-light fusion image to the fusion processing module; puts the three-light fusion image into a queue in sequence according to the ID sequence of the image frames formed by the three-light fusion image, and encodes them in sequence according to the queue sequence to obtain the encoded three-light fusion image; and sends the encoded three-light fusion image to the second processing module; S1.4: The second processing module sends the encoded three-light fusion image to the splicing unit via Ethernet using the UDP protocol.

[0014] Preferably, the specific method of judging whether the three-light image meets the requirements in step S1.2 is as follows: using the visible light image to identify the target and identifying the corresponding position of the target in the infrared image that is aligned with it, and performing adaptive threshold segmentation on the corresponding area in the infrared image; determining the maximum inscribed rectangle in the irregular area after image segmentation; determining the number of point clouds within the maximum inscribed rectangle in the laser point cloud image that is aligned with it, and judging whether the number of point clouds meets the requirements based on prior knowledge, and if so, sending the three-light image target frame that meets the requirements to the second processing module.

[0015] The beneficial effects of the present invention are as follows: by setting a splicing unit and four identical and distributed three-light fusion sensing units; the three-light fusion sensing unit includes an infrared thermal imager, a visible light camera, a laser radar, and a fusion module; the fusion module fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the laser radar, and sends the fused three-light fusion image data to the splicing unit through Ethernet, and the splicing unit splices the three-light fusion images sent by the four-way fusion module to obtain a panoramic image. The device realizes panoramic display of the scene through the fusion of three-light data, thereby realizing day and night monitoring of the scene and improving the accuracy of panoramic image target display. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall structure of the splicing device of the present invention; Figure 2 It is a schematic diagram of the internal structure of the fusion module and the splicing module of the present invention; Figure 3 is a schematic diagram of the image field of view during the stitching process of the present invention; Figure 4 It is a schematic flow chart of the splicing method of the present invention. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0018] like Figure 1 As shown, the present invention discloses a distributed three-light real-time splicing device, including: a splicing unit, four identical and distributed three-light fusion sensing units; the three-light fusion sensing unit includes an infrared thermal imager, a visible light camera, a laser radar, a fusion module, and a display unit; the fusion module performs fusion processing on the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the laser radar, and sends the fused three-light fusion image data to the splicing unit through Ethernet using the UDP protocol, and the splicing unit splices the three-light fusion images sent by the four-way fusion modules to obtain a panoramic image. The display unit displays the panoramic image, and can realize three modes: single-frame display, 180° up and down, and panoramic display.

[0019] like Figure 2 As shown, the fusion module includes: The first processing module pre-processes the images collected by the infrared thermal imager and the images collected by the visible light camera, and sends the pre-processed image data to the fusion processing module and the multi-modal recognition module respectively; wherein the infrared thermal imager, the visible light camera and the first processing module transmit image data via a communication interface, and the communication interface may specifically be a Camearlink interface; the first processing module uses an FPGA chip; The second processing module pre-processes the point cloud image collected by the laser radar; sends the pre-processed image data to the fusion processing module and the multimodal recognition module respectively; superimposes the three-light image target frame on the three-light fusion image, and sends the superimposed three-light fusion image to the fusion processing module; sends the encoded three-light fusion image to the splicing unit via Ethernet using the UDP protocol; the laser radar and the second processing module transmit point cloud data via Ethernet; the second processing module uses a SWITCH chip with an integrated PHY chip or is composed of a SWITCH chip with an external PHY chip.

[0020] The fusion processing module fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the laser radar, and sends the obtained three-light fusion image to the second processing module; according to the sequence of the image frame IDs formed by the three-light fusion image, they are sequentially put into a queue, and sequentially encoded according to the queue sequence to obtain the encoded three-light fusion image; and the encoded three-light fusion image is sent to the second processing module; The multimodal recognition module performs multimodal recognition processing on the image data collected by the infrared thermal imager, the image data collected by the visible light camera, and the point cloud data collected by the laser radar, determines whether the three-light images meet the requirements, and sends the three-light image target frames that meet the requirements to the second processing module; Both the fusion processing module and the multimodal recognition module use high-performance processors with specialized artificial intelligence application processing functions, specifically the RK3588 chip.

[0021] The output end of the laser radar is connected to the input end of the second processing module, the output end of the second processing module is connected to the input end of the splicing unit, the second processing module is connected to the fusion processing module and the multimodal recognition module, the fusion processing module and the multimodal recognition module are interconnected and respectively connected to the first processing module; the first processing module is connected to the infrared thermal imager, the visible light camera, and the splicing unit.

[0022] The splicing unit sends a synchronization signal to the four fusion modules through the RS422 serial port. The four fusion modules send the synchronization signal to the infrared thermal imager, the visible light camera and the laser radar through the RS422 serial port respectively. The splicing unit and the four fusion modules transmit communication and control instructions through CAN communication. The splicing unit includes: a receiving module and a processing module; the receiving module decodes and receives the encoded three-light fusion image and sends it to the processing module; the processing module sequentially performs numbering processing, image edge smoothing and filtering processing, and splicing processing on the received four-way three-light fusion images to obtain a panoramic image. Among them, the splicing processing adopts the texture splicing method.

[0023] like Figure 3 As shown in the figure, in the image stitching process, the positions of the four-way three-light fusion sensing units are set to be 1.5m apart from each other and 1.8m apart from each other. The processing module numbers the four-way three-light fusion images according to the IP address, and the specific numbers can be: 192.168.127.100, 192.168.127.110, 192.168.127.120, 192.168.127.130, corresponding to one-way three-light fusion sensing unit, respectively. At this time, the horizontal field of view of the three-light fusion image is 104°. Due to the existence of overlapping areas as shown in the shadow, when stitching and calibrating the three-light fusion images, reference objects are found in the overlapping areas. In the actual measurement scene, the reference objects can be selected: island coastline, the boundary between the sky and the sea, etc. To ensure the calibration effect, the distance of the reference object needs to be greater than 15m. The processing module performs operations such as translation and scaling on adjacent three-light fusion images according to the reference object, so that the image transition between adjacent fields of view is smooth, thereby obtaining a 360° panoramic image.

[0024] In actual application, the three-light fusion image can be translated, zoomed, and other operations performed by only adjusting the system parameters of the corresponding three-light fusion sensing unit to control the up and down position, left and right position, image zoom ratio, and stretching ratio of the corresponding three-light fusion sensing unit. By adjusting the parameters, the panoramic stitching of the distributed camera can be completed.

[0025] like Figure 4 As shown, a distributed three-light real-time splicing method includes the following steps: S1: The fusion module in the three-light fusion perception unit fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the lidar, and sends the fused three-light fusion image data to the splicing unit via Ethernet; S2: The stitching unit stitches the three-light fusion images sent by the four-way fusion module by using texture stitching to obtain a panoramic image.

[0026] Preferably, the step S1 specifically comprises: S1.1: The first processing module performs image smoothing, filtering, and distortion correction processing on the images collected by the infrared thermal imager and the visible light camera; the second processing module performs point cloud filtering and distortion correction processing on the point cloud image collected by the laser radar; the first processing module and the second processing module send the processed image data to the fusion processing module and the multimodal recognition module respectively; S1.2: The fusion processing module fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the lidar, and sends the obtained three-light fusion image to the second processing module; The multimodal recognition module performs multimodal recognition processing on the image data collected by the infrared thermal imager, the image data collected by the visible light camera, and the point cloud data collected by the laser radar, determines whether the three-light images meet the requirements, and sends the three-light image target frames that meet the requirements to the second processing module; S1.3: The second processing module superimposes the three-light image target frame on the three-light fusion image, and sends the superimposed three-light fusion image to the fusion processing module; the fusion processing module sequentially puts the image frames into a queue according to the ID sequence of the image frames formed by the three-light fusion image, and encodes them in sequence according to the queue sequence to obtain the encoded three-light fusion image; the encoded three-light fusion image is sent to the second processing module; S1.4: The second processing module sends the encoded three-light fusion image to the splicing unit via Ethernet using the UDP protocol.

[0027] Preferably, the specific method of judging whether the three-light image meets the requirements in step S1.2 is as follows: using the visible light image to identify the target and identifying the corresponding position of the target in the infrared image that is aligned with it, and performing adaptive threshold segmentation on the corresponding area in the infrared image; determining the maximum inscribed rectangle in the irregular area after image segmentation; determining the number of point clouds within the maximum inscribed rectangle in the laser point cloud image that is aligned with it, and judging whether the number of point clouds meets the requirements based on prior knowledge, and if so, sending the three-light image target frame that meets the requirements to the second processing module.

[0028] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

Claims

1. A distributed three-light real-time splicing device, characterized in that: include: A stitching unit, four identical three-light fusion sensing units that are distributed; the three-light fusion sensing unit includes an infrared thermal imager, a visible light camera, a laser radar, and a fusion module; the fusion module fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the laser radar, and sends the fused three-light fusion image data to the stitching unit via Ethernet; the stitching unit stitches the three-light fusion images sent by the four-way fusion modules to obtain a panoramic image.

2. The distributed three-light real-time splicing device according to claim 1, characterized in that: The fusion module includes: The first processing module pre-processes the images collected by the infrared thermal imager and the images collected by the visible light camera, and sends the pre-processed image data to the fusion processing module and the multi-modal recognition module respectively; The second processing module pre-processes the point cloud image collected by the laser radar; sends the pre-processed image data to the fusion processing module and the multimodal recognition module respectively; superimposes the three-light image target frame on the three-light fusion image, and sends the superimposed three-light fusion image to the fusion processing module; and sends the encoded three-light fusion image to the splicing unit via Ethernet using the UDP protocol; The fusion processing module fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the laser radar, and sends the obtained three-light fusion image to the second processing module; according to the sequence of the image frame IDs formed by the three-light fusion image, they are sequentially put into a queue, and sequentially encoded according to the queue sequence to obtain the encoded three-light fusion image; and the encoded three-light fusion image is sent to the second processing module; The multimodal recognition module performs multimodal recognition processing on the image data collected by the infrared thermal imager, the image data collected by the visible light camera, and the point cloud data collected by the laser radar, determines whether the three-light images meet the requirements, and sends the three-light image target frames that meet the requirements to the second processing module; The output end of the laser radar is connected to the input end of the second processing module, the output end of the second processing module is connected to the input end of the splicing unit, the second processing module is connected to the fusion processing module and the multimodal recognition module, the fusion processing module and the multimodal recognition module are interconnected and respectively connected to the first processing module; the first processing module is connected to the infrared thermal imager, the visible light camera, and the splicing unit.

3. The distributed three-light real-time splicing device according to claim 1, characterized in that: The splicing unit comprises: The receiving module decodes and receives the encoded three-light fusion image and sends it to the processing module; The processing module sequentially performs numbering processing, image edge smoothing and filtering processing, and splicing processing on the received four-way three-light fusion images to obtain a panoramic image.

4. The distributed three-light real-time splicing device according to claim 3, characterized in that: The splicing process adopts a texture splicing method.

5. The distributed three-light real-time splicing device according to claim 2, characterized in that: The infrared thermal imager, the visible light camera and the first processing module transmit image data via a communication interface; the laser radar and the second processing module transmit point cloud data via Ethernet.

6. The distributed three-light real-time splicing device according to claim 2, characterized in that: The splicing unit sends a synchronization signal to the four fusion modules, and the four fusion modules send the synchronization signal to the infrared thermal imager, the visible light camera, and the laser radar respectively.

7. The distributed three-light real-time splicing device according to claim 1, characterized in that: Also includes: The display unit displays the panoramic image; including three modes: single-frame display, up and down 180°, and panoramic display.

8. A distributed three-light real-time splicing method, using the distributed three-light real-time splicing device according to any one of claims 1 to 7, characterized in that: The following steps are involved: S1: The fusion module in the three-light fusion perception unit fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the lidar, and sends the fused three-light fusion image data to the splicing unit via Ethernet; S2: The stitching unit sequentially performs numbering processing, image edge smoothing and filtering processing, and stitching processing on the three-light fusion images sent by the four-way fusion module by means of texture stitching to obtain a panoramic image.

9. The distributed three-light real-time splicing method according to claim 8, characterized in that: The step S1 is specifically as follows: S1.1: The first processing module performs image smoothing, filtering, and distortion correction processing on the images collected by the infrared thermal imager and the visible light camera; the second processing module performs point cloud filtering and distortion correction processing on the point cloud image collected by the laser radar; the first processing module and the second processing module send the processed image data to the fusion processing module and the multimodal recognition module respectively; S1.2: The fusion processing module fuses the image collected by the infrared thermal imager, the image collected by the visible light camera, and the point cloud image collected by the lidar, and sends the obtained three-light fusion image to the second processing module; The multimodal recognition module performs multimodal recognition processing on the image data collected by the infrared thermal imager, the image data collected by the visible light camera, and the point cloud data collected by the laser radar, determines whether the three-light images meet the requirements, and sends the three-light image target frames that meet the requirements to the second processing module; S1.3: The second processing module superimposes the three-light image target frame on the three-light fusion image, and sends the superimposed three-light fusion image to the fusion processing module; puts the three-light fusion image into a queue in sequence according to the ID sequence of the image frames formed by the three-light fusion image, and encodes them in sequence according to the queue sequence to obtain the encoded three-light fusion image; and sends the encoded three-light fusion image to the second processing module; S1.4: The second processing module sends the encoded three-light fusion image to the splicing unit via Ethernet using the UDP protocol.

10. The distributed three-light real-time splicing method according to claim 9, characterized in that: The specific steps of judging whether the three-light image meets the requirements in step S1.2 are as follows: using the visible light image to identify the target and identifying the corresponding position of the target in the infrared image registered with it, and performing adaptive threshold segmentation on the corresponding area in the infrared image; determining the maximum inscribed rectangle within the irregular shape area after image segmentation; The number of point clouds within the maximum inscribed rectangular area is determined in the laser point cloud image that is aligned with it, and whether the number of point clouds meets the requirements is judged based on prior knowledge. If so, the three-light image target frame that meets the requirements is sent to the second processing module.

Images