Target identification system based on multiband binocular camera

Abstract

The invention relates to a target recognition system based on a multiband binocular camera, belongs to the technical field of target recognition, and solves the problems of inaccurate target recognition and low recognition efficiency in the prior art. The system comprises; a multi-band binocular camera, an edge calculation unit and a back-end calculation unit; the multiband binocular camera comprises two camera modules, and each camera module comprises an infrared camera and a low-light camera; the multiband binocular camera sends two paths of infrared images and two paths of low-light images shot by the multiband binocular camera to the edge calculation unit and the back-end calculation unit, and the edge calculation unit performs target detection based on the two paths of infrared images and the two paths of low-light images to obtain a target detection result; sending the target detection result to a back-end calculation unit; the rear-end calculation unit performs binocular recognition based on the two paths of infrared images and the two paths of low-light images to obtain a binocular recognition result; and the back-end calculation unit fuses the target detection result and the binocular recognition result to obtain a target recognition result.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and in particular to a target recognition system based on a multi-band binocular camera. Background Art

[0002] In video surveillance, detecting, identifying and automatically alarming moving targets have always been a hot research issue, and it has important applications in vehicle safety, security monitoring, and robotics. In the past decade, there have been many mature algorithms for moving target detection technology, but there are still many problems and difficulties. In good lighting conditions, only visible light cameras are needed to obtain images with rich texture information, but in rainy days, heavy fog, low light at night, and external sudden strong light interference, the target features in the visible light camera image are not obvious, and the detection results are inaccurate. Summary of the invention

[0003] In view of the above analysis, an embodiment of the present invention aims to provide a target recognition system based on a multi-band binocular camera to solve the problems of inaccurate recognition and low recognition efficiency of existing mobile targets.

[0004] On the one hand, an embodiment of the present invention provides a target recognition system based on a multi-band binocular camera, comprising: a multi-band binocular camera, an edge computing unit and a backend computing unit;

[0005] The multi-band binocular camera includes two camera modules, each of which includes an infrared camera and a low-light camera; the multi-band binocular camera sends the captured two infrared images and two low-light images to the edge computing unit and the back-end computing unit.

[0006] The edge computing unit performs target detection based on the two infrared images and the two low-light images to obtain a target detection result; and sends the target detection result to the back-end computing unit;

[0007] The back-end computing unit performs binocular recognition based on the two infrared images and the two low-light images to obtain a binocular recognition result;

[0008] The back-end computing unit fuses the target detection result and the binocular recognition result to obtain the target recognition result.

[0009] Based on the further improvement of the above system, the edge computing unit performs target detection based on the two infrared images and the two low-light images to obtain a target detection result, including:

[0010] Based on the deep learning method, target detection is performed on the two infrared images and the two low-light images respectively, and target detection results of each infrared image and each low-light image are obtained respectively;

[0011] The detection confidence of each low-light camera is determined based on each low-light image. If the detection confidence of both low-light images is high, the detection result of the first low-light image and the detection result of the first infrared image are combined to obtain a first target detection result; the detection result of the second low-light image and the detection result of the second infrared image are combined to obtain a second target detection result;

[0012] If there is a low-light image with low detection confidence, the union of the detection result of the low-light image with high detection confidence and the detection result of the corresponding infrared image is taken as the first target detection result, and the union of the detection result of the low-light image with low detection confidence and the detection result of the corresponding infrared image is taken as the second target detection result;

[0013] If the detection confidence of the two low-light images is low, the detection result of the first infrared image is used as the first target detection result, and the detection result of the second infrared image is used as the second target detection result.

[0014] Based on the further improvement of the above system, the detection result of the first low-light image and the detection result of the first infrared image are combined to obtain the first target detection result in the following manner:

[0015] For each target frame in the detection result of the first infrared image, determine whether there is a target frame in the detection result of the first low-light image whose overlap with the target frame is greater than a third threshold; if not, add the target frame in the detection result of the first infrared image to the detection result of the first low-light image to obtain the first target detection result.

[0016] Based on the further improvement of the above system, the detection confidence of each low-light camera is judged based on each low-light image, including:

[0017] For each low-light camera, take multiple consecutive low-light images before the current low-light image, and determine whether the average brightness of the current low-light image and the multiple consecutive low-light images before the current low-light image is less than a first threshold. If so, the detection confidence of the low-light camera is low; otherwise, the detection confidence of the low-light camera is high.

[0018] Based on the further improvement of the above system, the back-end computing unit performs binocular recognition based on the two infrared images and the two low-light images to obtain binocular recognition results, including:

[0019] Perform binocular recognition based on the first low-light image and the second low-light image to obtain a first binocular recognition result;

[0020] Perform binocular recognition based on the first infrared image and the second infrared image to obtain a second binocular recognition result;

[0021] The first infrared image is fused with the first low-light image to obtain a first fused image; the second infrared image is fused with the second low-light image to obtain a second fused image; binocular recognition is performed based on the first fused image and the second fused image to obtain a third binocular recognition result;

[0022] Based on the first binocular recognition result, the second binocular recognition result and the third binocular recognition result, a final binocular recognition result is determined according to the validity and detection confidence of each camera.

[0023] Based on the further improvement of the above system, the following method is used to determine the effectiveness of each camera:

[0024] For each camera, the current image and multiple consecutive images taken before the current image are obtained, and the pixel similarity of each image is calculated. If the pixel similarity of each image is greater than the second threshold, the camera is invalid; otherwise, the camera is valid.

[0025] Based on the further improvement of the above system, based on the first binocular recognition result, the second binocular recognition result and the third binocular recognition result, according to the effectiveness and detection confidence of each camera, the final binocular recognition result is determined, including:

[0026] If all cameras are valid and the detection confidence of the two low-light cameras is high, the final binocular recognition result is the union of the first binocular recognition result and the second binocular recognition result; if the detection confidence of one of the low-light cameras is low, the final binocular recognition result is the union of the second binocular recognition result and the third binocular recognition result;

[0027] If one of the low-light cameras fails and both infrared cameras are valid, the final binocular recognition result is the second binocular recognition result;

[0028] If one of the infrared cameras fails and both low-light cameras are valid, the final binocular recognition result is the first binocular recognition result.

[0029] Based on the further improvement of the above system, the union of the first binocular recognition result and the second binocular recognition result is obtained in the following manner:

[0030] Taking the connected domain in which the difference of pixel values ​​in the first binocular recognition result is within the range of the fourth threshold as a contour, outlining the contour of the object in the first binocular recognition result;

[0031] Taking the connected domain whose pixel value difference in the second binocular recognition result is within the range of the fourth threshold as a contour, outlining the contour of the object in the second binocular recognition result;

[0032] For each object contour in the second binocular recognition result, if there is no object contour in the first binocular recognition result that overlaps with it by more than the third threshold, and the difference between the maximum pixel value of the object contour in the second binocular recognition result and the minimum pixel value in the corresponding area of ​​the object contour in the first binocular recognition result is greater than the fifth threshold, then the object contour in the second binocular recognition result is mapped to the first target recognition result; and the union of the first binocular recognition result and the second binocular recognition result is obtained.

[0033] Based on the further improvement of the above system, the back-end computing unit fuses the target detection result and the binocular recognition result to obtain the target recognition result, including:

[0034] The back-end computing unit obtains an image corresponding to the target detection result sent by the edge computing unit;

[0035] Calculate the target movement speed according to the target recognition result before the image; calculate the position of the target in the latest binocular recognition image based on the target movement speed;

[0036] The target recognition result is obtained according to the position of the target in the latest binocular recognition image and the latest binocular recognition result.

[0037] Based on the further improvement of the above system, the moving speed of the target is calculated according to the following formula:

[0038]

[0039]

[0040] The position p(x,y) of the target in the latest binocular recognition result is calculated according to the formula p(x,y)=(tΔx,tΔy); where x i and i They respectively represent the x-axis coordinate and y-axis coordinate of the target in the target recognition result of the i-th image before the image corresponding to the target recognition result, t represents the time difference between the shooting time of the image corresponding to the target detection result and the shooting time of the image corresponding to the latest binocular recognition result, n represents the n target recognition result images before the image, Δx represents the moving speed of the target on the x-axis, and Δy represents the moving speed of the target on the y-axis.

[0041] Compared with the prior art, the multi-band binocular camera of the present invention is equipped with a low-light camera and an infrared camera in each camera module for image capture, which organically combines target detection and binocular depth recognition, so that relatively clear images can be obtained even in harsh environments (low illumination, strong light, rain, snow and fog), thereby improving the reliability and accuracy of target recognition. By setting an edge computing unit in the multi-band binocular camera, the back-end computing pressure is reduced, and the reliability and efficiency of recognition are improved.

[0042] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0044] Figure 1 The block diagram of the target recognition system based on the multi-band binocular camera according to the embodiment of the present invention. DETAILED DESCRIPTION

[0045] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0046] A specific embodiment of the present invention discloses a target recognition system based on a multi-band binocular camera. Figure 1 As shown, the system includes: a multi-band binocular camera, an edge computing unit and a back-end computing unit.

[0047] The multi-band binocular camera consists of two camera modules, each of which includes an infrared camera and a low-light camera.

[0048] The multi-band binocular camera sends the captured two-way infrared images and two-way low-light images to the edge computing unit and the back-end computing unit.

[0049] The edge computing unit performs target detection based on the two infrared images and the two low-light images to obtain a target detection result; and sends the target detection result to the back-end computing unit;

[0050] The back-end computing unit performs binocular recognition based on the two infrared images and the two low-light images to obtain a binocular recognition result;

[0051] The back-end computing unit fuses the target detection result and the binocular recognition result to obtain the target recognition result.

[0052] During implementation, the edge computing unit is set in the multi-band binocular camera. The images captured by the multi-band binocular camera are sent to the edge computing unit and the back-end computing unit at the same time for target detection and binocular recognition. The edge computing unit sends the detection result to the back-end computing unit, and the back-end computing unit fuses the target detection result and the binocular recognition result to obtain the target recognition result.

[0053] Compared with the prior art, the multi-band binocular camera of the present invention is equipped with a low-light camera and an infrared camera in each camera module for image capture, which organically combines target detection and binocular depth recognition, so that relatively clear images can be obtained even in harsh environments (low illumination, strong light, rain, snow and fog), thereby improving the reliability and accuracy of target recognition. By setting an edge computing unit in the multi-band binocular camera, the back-end computing pressure is reduced, and the reliability and efficiency of recognition are improved.

[0054] During implementation, the edge computing unit can use NPU or GPU, and the backend computing unit can use FPGA. The image captured by the camera is transmitted to the camera's CPU through interfaces such as V4l2 by DMA, a copy is made in the CPU, and then the two copies of the data are simultaneously transmitted to the NPU and FPGA by DMA.

[0055] For the convenience of explanation, the low-light camera in the first camera module is represented by A1, and the image captured is the first low-light image. The infrared camera is represented by A2, and the image captured is the first infrared image. The low-light camera in the second camera module is represented by B1, and the image captured is the second low-light image. The infrared camera is represented by B2, and the image captured is the second infrared image. The two camera modules continuously capture images and send them to the edge computing unit and the back-end computing unit.

[0056] Specifically, the edge computing unit performs target detection based on the two infrared images and the two low-light images to obtain a target detection result, including:

[0057] Based on the deep learning method, target detection is performed on the two infrared images and the two low-light images respectively, and target detection results of each infrared image and each low-light image are obtained respectively;

[0058] The detection confidence of each low-light camera is determined based on each low-light image. If the detection confidence of both low-light images is high, the detection result of the first low-light image and the detection result of the first infrared image are combined to obtain a first target detection result; the detection result of the second low-light image and the detection result of the second infrared image are combined to obtain a second target detection result;

[0059] If there is a low-light image with low detection confidence, the union of the detection result of the low-light image with high detection confidence and the detection result of the corresponding infrared image is taken as the first target detection result, and the union of the detection result of the low-light image with low detection confidence and the detection result of the corresponding infrared image is taken as the second target detection result;

[0060] If the detection confidence of the two low-light images is low, the detection result of the first infrared image is used as the first target detection result, and the detection result of the second infrared image is used as the second target detection result.

[0061] Target detection is to detect moving objects contained in the image, such as pedestrians or vehicles. Target detection can be performed using existing deep learning target detection methods. Detected targets and their corresponding types are marked in the image using detection boxes to obtain target detection results. During implementation, existing deep learning methods can be used to perform target detection on each image to obtain detection results. The detection results that are ultimately output by the edge computing unit are determined based on the detection confidence of the low-light camera.

[0062] Specifically, the detection confidence of each low-light camera is determined based on each low-light image, including:

[0063] For each low-light camera, take multiple consecutive low-light images before the current low-light image, and determine whether the average brightness of the current low-light image and the multiple consecutive low-light images before the current low-light image is less than a first threshold. If so, the detection confidence of the low-light camera is low; otherwise, the detection confidence of the low-light camera is high.

[0064] When the light intensity is very low, the detection of the low-light camera is inaccurate, so the detection confidence of the low-light camera can be judged by the image brightness information. For example, for the low-light camera A1, take its latest multiple consecutive images, and for each image, calculate its average brightness. In the YUV format image, the Y component represents the brightness information of the image. The average brightness of each image can be obtained by adding and averaging the brightness of each pixel in the image. If the average brightness of multiple consecutive images is less than the first threshold, it means that the lighting conditions are particularly poor at this time, and the low-light image recognition is inaccurate. It is judged that the detection confidence of the low-light camera A1 is low, otherwise the detection confidence of the low-light camera A1 is high.

[0065] If the detection confidences of the two low-light cameras are both high, the detection results of the first low-light image and the first infrared image are combined to obtain a first target detection result; the detection results of the second low-light image and the second infrared image are combined to obtain a second target detection result.

[0066] Specifically, the detection result of the first low-light image and the detection result of the first infrared image are combined to obtain the first target detection result in the following manner:

[0067] For each target frame in the detection result of the first infrared image, determine whether there is a target frame in the detection result of the first low-light image whose overlap with the target frame is greater than a third threshold; if not, add the target frame in the detection result of the first infrared image to the detection result of the first low-light image to obtain the first target detection result.

[0068] In implementation, if the target area overlap in the two images is greater than the third threshold, they are considered to be the same target, otherwise they are different targets, and the target in the detection result of the infrared image that is not in the detection result of the low-light image is added to the detection result of the low-light image, that is, marked at the corresponding position of the low-light image, to obtain the merged target detection result. In implementation, the third threshold can be, for example, 95%.

[0069] If there is a low-light image with low detection confidence, the union of the detection results of the low-light image with high detection confidence and the corresponding infrared image is taken as the first target detection result, and the union of the detection results of the low-light image with low detection confidence and the corresponding infrared image is taken as the second target detection result.

[0070] If the detection confidence of the two low-light images is low, the detection result of the first infrared image is used as the first target detection result, and the detection result of the second infrared image is used as the second target detection result.

[0071] During implementation, the first threshold can be obtained in the following manner: when the light gradually decreases to, noise begins to appear in the camera image data, and binocular calculation is performed on the image during the process of decreasing light. When the accuracy decreases to 3%, the average Y component of the image YUV at this time is recorded as mLux, which is the first threshold.

[0072] The edge computing unit sends the first detection result and the second detection result to the backend computing unit.

[0073] During implementation, in actual scenes, camera failures may occur due to sudden changes in light, such as blindness caused by entering and exiting tunnels, opposite-side high beams, and light refraction. When the edge computing unit detects that the average brightness of the low-light camera is low for a continuous period of time, it can control the turning on of the infrared fill light for light compensation, thereby improving the detection confidence of the low-light camera. For example, if 250 consecutive frames of images are detected, and the average brightness of each frame is less than mLux, the infrared fill light is controlled to be turned on for optical fiber compensation. After turning on the infrared fill light for a period of time, if it is detected that the average brightness of the low-light camera is high for a continuous period of time, for example, the average brightness of each image of 250 consecutive frames of images is greater than mLux+20, it means that the light intensity has increased at this time, and it is judged that the low-light camera is effective at this time, and the infrared fill light is turned off.

[0074] The back-end computing unit performs binocular recognition based on the two infrared images and the two low-light images to obtain a binocular recognition result, including:

[0075] Perform binocular recognition based on the first low-light image and the second low-light image to obtain a first binocular recognition result;

[0076] Perform binocular recognition based on the first infrared image and the second infrared image to obtain a second binocular recognition result;

[0077] The first infrared image is fused with the first low-light image to obtain a first fused image; the second infrared image is fused with the second low-light image to obtain a second fused image; binocular recognition is performed based on the first fused image and the second fused image to obtain a third binocular recognition result;

[0078] Based on the first binocular recognition result, the second binocular recognition result and the third binocular recognition result, a final binocular recognition result is determined according to the validity and detection confidence of each camera.

[0079] During implementation, the backend computing unit uses the same method as the edge computing unit to judge the confidence level of each camera.

[0080] During implementation, the existing image fusion technology can be used to fuse the two images taken by the first camera module to obtain a first fused image, and the two images taken by the second camera module can be fused to obtain a second fused image, for example, using the RPN-Nest deep learning model for image fusion.

[0081] In implementation, image binocular recognition is to calculate the depth of each pixel in the image. In implementation, the depth map can be calculated using the existing binocular recognition algorithm.

[0082] Since camera failure may occur due to camera occlusion, it is necessary to determine whether each camera is valid. Specifically, the following methods are used to determine the validity of each camera:

[0083] For each camera, the current image and multiple consecutive images taken before the current image are obtained, and the pixel similarity of each image is calculated. If the pixel similarity of each image is greater than the second threshold, the camera is invalid; otherwise, the camera is valid.

[0084] For example, for the low-light camera A1, take its latest multiple consecutive images and calculate the number of identical pixel values ​​in each image, such as the number of pixels with a pixel value and the number of pixels with a pixel value of b. The maximum number of identical pixel values ​​is the pixel similarity value corresponding to the image. If the pixel similarity of each image in the multiple consecutive images is greater than the second threshold, it means that the camera is blocked, and the low-light camera A1 is invalid. During implementation, 25 consecutive images can be taken. The second threshold can be 85%. The same process is used to determine whether the infrared camera A2, the low-light camera B1 and the infrared camera B2 are valid. The same method as above is used to determine whether each low-light camera is valid. Based on the first binocular recognition result, the second binocular recognition result and the third binocular recognition result, the final binocular recognition result is determined according to the effectiveness and detection confidence of each camera, specifically including:

[0085] If all cameras are valid and the detection confidence of the two low-light cameras is high, the final binocular recognition result is the union of the first binocular recognition result and the second binocular recognition result; if the detection confidence of one of the low-light cameras is low, the final binocular recognition result is the union of the second binocular recognition result and the third binocular recognition result;

[0086] If one of the low-light cameras fails and both infrared cameras are valid, the final binocular recognition result is the second binocular recognition result;

[0087] If one of the infrared cameras fails and both low-light cameras are valid, the final binocular recognition result is the first binocular recognition result.

[0088] If all four cameras are valid and the detection confidence of the two low-light cameras is high, the final binocular recognition result is the union of the first recognition result and the second recognition result. If there is a low-light camera with low detection confidence, the result of low-light binocular recognition is the union of the second binocular recognition result and the third binocular recognition result as the final binocular recognition result. Since the third recognition result is obtained based on the fusion image of low-light and infrared, by fusing the first recognition result or the second recognition result with the third recognition result, the binocular recognition result of the low-light image and the binocular recognition result of the infrared image are fused, the recognition is more accurate, the recognition ability in harsh environments is improved, and the system error caused by single-band recognition is reduced.

[0089] The result of binocular recognition is a depth map, which is a single-channel grayscale image in which the smaller the pixel value, the closer the object is to the camera, and the larger the pixel value, the farther the object is from the camera.

[0090] During implementation, the union of the first binocular recognition result and the second binocular recognition result is obtained in the following manner:

[0091] Taking the connected domain in which the difference of pixel values ​​in the first binocular recognition result is within the range of the fourth threshold as a contour, outlining the contour of the object in the first binocular recognition result;

[0092] Taking the connected domain whose pixel value difference in the second binocular recognition result is within the range of the fourth threshold as a contour, outlining the contour of the object in the second binocular recognition result;

[0093] For each object contour in the second binocular recognition result, if there is no object contour in the first binocular recognition result that overlaps with it by more than the third threshold, and the difference between the maximum pixel value of the object contour in the second binocular recognition result and the minimum pixel value in the corresponding area of ​​the object contour in the first binocular recognition result is greater than the fifth threshold, then the object contour in the second binocular recognition result is mapped to the first target recognition result; and the union of the first binocular recognition result and the second binocular recognition result is obtained.

[0094] For example, an object contour a is outlined in the depth map of the second binocular recognition result. In the depth map of the first recognition result, there is no object contour in which the overlap with the object contour a is greater than the third threshold, and the difference between the maximum pixel value in the object contour a and the minimum pixel value in the area corresponding to the object contour a in the depth map of the first binocular recognition result is greater than the fifth threshold, indicating that the object recognized in the second binocular recognition result is not recognized in the first binocular recognition result. Therefore, the object contour a is mapped to the first target recognition result, that is, the pixel value of the area corresponding to the object contour a in the depth map of the first binocular recognition result is modified to the pixel value of the object contour a.

[0095] If one of the low-light cameras fails, but both infrared cameras are valid, the final binocular recognition result is the second binocular recognition result. If the low-light camera fails, the infrared camera can still perform binocular depth detection, so that the target recognition system can operate normally. At this time, a low-light camera failure warning can be issued to facilitate timely repair and replacement.

[0096] If one of the infrared cameras fails, but both low-light cameras are valid, the final binocular recognition result is the first binocular recognition result. If the infrared camera fails, the low-light camera can still perform binocular depth detection, so that the target recognition system can operate normally. At this time, an infrared camera failure warning can be issued to facilitate timely repair and replacement. If the detection confidence of one low-light camera is low at this time, a prompt will be given that the image brightness of the low-light camera is too low.

[0097] The infrared camera and the low-light camera serve as redundancy for each other, thus achieving more reliable target detection results. When three or more cameras fail, the binocular recognition fails, and the binocular recognition fails is displayed, and no image is output.

[0098] During implementation, the back-end computing unit can perform binocular recognition calculations in parallel. Since FPGA uses pipeline operation, the computing speed is slower than that of the edge computing unit. Therefore, the captured image corresponding to the target detection result sent by the edge computing unit to FPGA does not correspond to the image of the binocular recognition result currently calculated by FPGA. For example, the edge computing unit sends the result calculated based on the 50th frame image, and FPGA currently calculates the binocular recognition result of the 60th frame image. Therefore, it is necessary to fuse the target detection result and the binocular recognition result to obtain the target recognition result.

[0099] Specifically, the back-end computing unit fuses the target detection result and the binocular recognition result to obtain the target recognition result, including:

[0100] The back-end computing unit obtains an image corresponding to the target detection result sent by the edge computing unit;

[0101] Calculate the target movement speed according to the target recognition result before the image; calculate the position of the target in the latest binocular recognition image based on the target movement speed;

[0102] The target recognition result is obtained according to the position of the target in the latest binocular recognition image and the latest binocular recognition result.

[0103] For example, the edge computing unit sends the target detection result calculated based on the 50th frame image, and the FPGA currently calculates the binocular recognition result of the 60th frame image. First, according to the position of the target in the binocular recognition results of multiple images before the 50th frame, the average moving speed of the target is calculated. According to the moving speed and the time difference, that is, the time difference between the 50th frame and the 60th frame, the position of the target in the 60th frame image is obtained. It should be noted that when the position of the target is based on the binocular recognition results of the 50th frame and the 50th frame, if the overlap of the target area in two consecutive frames is greater than the third threshold, it is considered to be the same target.

[0104] For example, for target a, the positions of its target frame in the 5 frames before the 50th frame are: (x 1, y1)、(x 2, y2), (x3,y3), (x4,y3), (x5,y5), calculate the target's moving speed according to the following formula:

[0105]

[0106]

[0107] The position p(x,y) of the target in the latest binocular recognition result is calculated according to the formula p(x,y)=(tΔx,tΔy); where x i and i They respectively represent the x-axis coordinate and y-axis coordinate of the target in the target recognition result of the i-th image before the image corresponding to the target recognition result, t represents the time difference between the shooting time of the image corresponding to the target detection result and the shooting time of the image corresponding to the latest binocular recognition result, n represents the n target recognition result images before the image, Δx represents the moving speed of the target on the x-axis, and Δy represents the moving speed of the target on the y-axis.

[0108] During implementation, the coordinate system is an image coordinate system, for example, the first point in the upper left corner of the image is taken as the coordinate origin, the horizontal rightward direction is the positive direction of the x-axis, and the vertical downward direction is the positive direction of the y-axis.

[0109] It should be noted that the x-axis coordinate and y-axis coordinate of the target in the target recognition result refer to the coordinates of the center point of the target area.

[0110] After calculating the position of the target in the latest binocular recognition image, the recognition result of the target can be obtained based on the binocular recognition result, that is, the detection frame of the target in the image and the distance of the target. During implementation, the back-end computing unit can take the first target detection result sent by the edge computing unit for fusion, and the second target detection result can be used as redundant data.

[0111] If the final binocular recognition result is the union of the first binocular recognition result and the second binocular recognition result, the target contour is marked on the first low-light image; if the final binocular recognition result is the union of the second binocular recognition result and the third binocular recognition result, the target contour is marked on the first fused image; if the final binocular recognition result is the second binocular recognition result; the target contour is marked on the first infrared image; if the final binocular recognition result is the first binocular recognition result, the target contour is marked on the image with a larger average brightness between the first red low-light image and the second low-light image.

[0112] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0113] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A target recognition system based on a multi-band binocular camera, characterized in that: Including; multi-band binocular camera, edge computing unit and back-end computing unit; The multi-band binocular camera includes two camera modules, each of which includes an infrared camera and a low-light camera; the multi-band binocular camera sends the captured two infrared images and two low-light images to the edge computing unit and the back-end computing unit. The edge computing unit performs target detection based on the two infrared images and the two low-light images to obtain a target detection result; and sends the target detection result to the back-end computing unit; The back-end computing unit performs binocular recognition based on the two infrared images and the two low-light images to obtain a binocular recognition result; The back-end computing unit fuses the target detection result and the binocular recognition result to obtain the target recognition result.

2. The target recognition system based on a multi-band binocular camera according to claim 1, characterized in that: The edge computing unit performs target detection based on the two infrared images and the two low-light images to obtain a target detection result, including: Based on the deep learning method, target detection is performed on the two infrared images and the two low-light images respectively, and target detection results of each infrared image and each low-light image are obtained respectively; The detection confidence of each low-light camera is determined based on each low-light image. If the detection confidence of both low-light images is high, the detection result of the first low-light image and the detection result of the first infrared image are combined to obtain a first target detection result; the detection result of the second low-light image and the detection result of the second infrared image are combined to obtain a second target detection result; If there is a low-light image with low detection confidence, the union of the detection result of the low-light image with high detection confidence and the detection result of the corresponding infrared image is taken as the first target detection result, and the union of the detection result of the low-light image with low detection confidence and the detection result of the corresponding infrared image is taken as the second target detection result; If the detection confidence of the two low-light images is low, the detection result of the first infrared image is used as the first target detection result, and the detection result of the second infrared image is used as the second target detection result.

3. The target recognition system based on a multi-band binocular camera according to claim 2, characterized in that: The detection result of the first low-light image and the detection result of the first infrared image are combined to obtain the first target detection result in the following manner: For each target frame in the detection result of the first infrared image, determine whether there is a target frame in the detection result of the first low-light image whose overlap with the target frame is greater than a third threshold; if not, add the target frame in the detection result of the first infrared image to the detection result of the first low-light image; The first target detection result is obtained.

4. The target recognition system based on a multi-band binocular camera according to claim 2, characterized in that: The detection confidence of each low-light camera is determined based on each low-light image, including: For each low-light camera, take multiple consecutive low-light images before the current low-light image, and determine whether the average brightness of the current low-light image and the multiple consecutive low-light images before the current low-light image is less than a first threshold. If so, the detection confidence of the low-light camera is low; otherwise, the detection confidence of the low-light camera is high.

5. The target recognition system based on a multi-band binocular camera according to claim 1, characterized in that: The back-end computing unit performs binocular recognition based on the two infrared images and the two low-light images to obtain a binocular recognition result, including: Perform binocular recognition based on the first low-light image and the second low-light image to obtain a first binocular recognition result; Perform binocular recognition based on the first infrared image and the second infrared image to obtain a second binocular recognition result; The first infrared image is fused with the first low-light image to obtain a first fused image; the second infrared image is fused with the second low-light image to obtain a second fused image; binocular recognition is performed based on the first fused image and the second fused image to obtain a third binocular recognition result; Based on the first binocular recognition result, the second binocular recognition result and the third binocular recognition result, a final binocular recognition result is determined according to the validity and detection confidence of each camera.

6. The target recognition system based on a multi-band binocular camera according to claim 5, characterized in that: The effectiveness of each camera is determined in the following way: For each camera, the current image and multiple consecutive images taken before the current image are obtained, and the pixel similarity of each image is calculated. If the pixel similarity of each image is greater than the second threshold, the camera is invalid; otherwise, the camera is valid.

7. The target recognition system based on a multi-band binocular camera according to claim 5, characterized in that: Based on the first binocular recognition result, the second binocular recognition result and the third binocular recognition result, the final binocular recognition result is determined according to the effectiveness and detection confidence of each camera, including: If all cameras are valid and the detection confidence of the two low-light cameras is high, the final binocular recognition result is the union of the first binocular recognition result and the second binocular recognition result; if the detection confidence of one of the low-light cameras is low, the final binocular recognition result is the union of the second binocular recognition result and the third binocular recognition result; If one of the low-light cameras fails and both infrared cameras are valid, the final binocular recognition result is the second binocular recognition result; If one of the infrared cameras fails and both low-light cameras are valid, the final binocular recognition result is the first binocular recognition result.

8. The target recognition system based on a multi-band binocular camera according to claim 7, characterized in that: The union of the first binocular recognition result and the second binocular recognition result is obtained in the following manner: Taking the connected domain in which the difference of pixel values ​​in the first binocular recognition result is within the range of the fourth threshold as a contour, outlining the contour of the object in the first binocular recognition result; Taking the connected domain whose pixel value difference in the second binocular recognition result is within the range of the fourth threshold as a contour, outlining the contour of the object in the second binocular recognition result; For each object contour in the second binocular recognition result, if there is no object contour in the first binocular recognition result that overlaps with it by more than the third threshold, and the difference between the maximum pixel value of the object contour in the second binocular recognition result and the minimum pixel value in the corresponding area of ​​the object contour in the first binocular recognition result is greater than the fifth threshold, then the object contour in the second binocular recognition result is mapped to the first target recognition result; A union of the first binocular recognition result and the second binocular recognition result is obtained.

9. The target recognition system based on a multi-band binocular camera according to claim 1, characterized in that: The back-end computing unit fuses the target detection result and the binocular recognition result to obtain the target recognition result, including: The back-end computing unit obtains an image corresponding to the target detection result sent by the edge computing unit; Calculate the target movement speed according to the target recognition result before the image; calculate the position of the target in the latest binocular recognition image based on the target movement speed; The target recognition result is obtained according to the position of the target in the latest binocular recognition image and the latest binocular recognition result.

10. The target recognition system based on a multi-band binocular camera according to claim 9, characterized in that: The target's moving speed is calculated according to the following formula: The position p(x,y) of the target in the latest binocular recognition result is calculated according to the formula p(x,y)=(tΔx,tΔy); where x i and i They respectively represent the x-axis coordinate and y-axis coordinate of the target in the target recognition result of the i-th image before the image corresponding to the target recognition result, t represents the time difference between the shooting time of the image corresponding to the target detection result and the shooting time of the image corresponding to the latest binocular recognition result, n represents the n target recognition result images before the image, Δx represents the moving speed of the target on the x-axis, and Δy represents the moving speed of the target on the y-axis.

Images