Dynamic target detection method and system based on multi-camera device

Abstract

The invention relates to a dynamic target detection method and system based on a multi-camera device. The dynamic target detection method is applied to a multi-camera device comprising a color camera, a low-light camera, an infrared camera and an event camera, and comprises the following steps: acquiring images shot by the color camera, the low-light camera and the infrared camera at a set moment, and respectively carrying out target detection, obtaining a target TC found by a color camera, a target TL found by a low-light camera and a target TI found by an infrared camera; obtaining a dynamic event graph TE in a visual field range found by an event camera at the set moment; and determining whether a dynamic target is found based on the target TC, the target TL and the contour coincidence degree of the target TI and the dynamic event graph TE. The method can be suitable for various illumination environments, eliminates the influence of the background on the detection algorithm, and improves the robustness degree of the algorithm for false target recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic target detection, and in particular to a dynamic target detection method and system based on a multi-camera device. Background Art

[0002] Object detection has a wide range of applications in security monitoring, field exploration, humanoid robots, autonomous driving, and other fields. In security systems, object detection can be used to monitor whether the target area has been invaded, as well as to detect specific targets and behaviors; in field exploration, object detection can detect specific animals and count their frequency of appearance and life patterns; on humanoid robots, object detection can detect the robot's potential work targets and prepare for subsequent intelligent analysis; in the field of autonomous driving, object detection is used to identify obstacles such as vehicles and pedestrians on the road to ensure driving safety.

[0003] At present, there are still many limitations in target detection. For example, the influence of lighting causes most target detection systems to be unable to work all day long; when the target and background features are highly overlapped, it is impossible to effectively distinguish the target from the background; it is easy to be deceived by false targets, resulting in misidentification, etc. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a dynamic target detection method and system based on a multi-camera device, which can be applicable to various lighting environments and eliminate the influence of the background on the detection algorithm, thereby improving the robustness of the algorithm for false target recognition.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a dynamic target detection method based on a multi-camera device, which is applied to a multi-camera device including a color camera, a low-light camera, an infrared camera, and an event camera, and includes the following steps:

[0006] Get the images taken by the color camera, low-light camera, and infrared camera at the set time and perform target detection respectively to obtain the target T found by the color camera C , target T detected by low-light camera L , and the target T detected by the infrared camera I ;

[0007] Get the dynamic event graph T within the field of view discovered by the event camera at the set time E ;

[0008] Based on the target T C Target T L , and target T I With dynamic event graph T E The degree of contour overlap determines whether a dynamic target is found.

[0009] Furthermore, the target T C Target T L , and target T I With dynamic event graph T E The degree of contour overlap determines whether a dynamic target is found, including:

[0010] Extract target T respectively C Target T L and target T I The contour feature map is obtained by C , Feature map B L And feature map B I ; Extract dynamic event graph T E The event position features are obtained by E ;

[0011] Calculate the feature map B separately C , Feature map B L , and feature map B I With feature map B E The degree of feature matching between them;

[0012] Whether a dynamic target is found is determined based on the calculated feature matching degree.

[0013] Furthermore, the feature matching degree is calculated by the following method:

[0014] The feature map B C , Feature map B L , and feature map B I The pixel value of the contour area is set to the first set value, and the pixel value of the remaining area is set to the second set value, and the feature map B' is obtained. c , feature map B' L , and feature map B' I ;

[0015] The feature map B' C , feature map B' L , and feature map B' I As a Mask, in the feature map B E Move up, count the number of pixels in the overlapped area whose pixel values ​​are all the first set value, and divide it by the feature map B E After the total number of pixels is calculated, normalization is performed to obtain the feature matching degree.

[0016] Furthermore, the extraction of dynamic event graph T E The event position features are obtained by E ,include:

[0017] The dynamic event graph T EThe pixel value of the event position is set to the first set value, and the pixel values ​​of the remaining areas are set to the second set value, and the feature map B is obtained. E .

[0018] Furthermore, the first setting value is 255, and the second setting value is 0.

[0019] Further, determining whether a dynamic target is found according to the calculated feature matching degree includes:

[0020] If the calculated matching degree of any feature is greater than the set threshold, it is considered that a dynamic target is found.

[0021] Furthermore, the color camera, low-light camera, infrared camera, and event camera have overlapping fields of view.

[0022] The present invention also provides a dynamic target detection system based on a multi-camera device, comprising:

[0023] Multi-camera setup, including color camera, low-light camera, infrared camera, and event camera;

[0024] The acquisition module is used to obtain the images taken by the color camera, the low-light camera, and the infrared camera at the set time, as well as the dynamic event map within the field of view discovered by the event camera at the set time. E ;

[0025] The target detection module is used to perform target detection on the images taken by the color camera, the low-light camera, and the infrared camera, and obtain the target T found by the color camera. C , target T detected by low-light camera L , and the target T detected by the infrared camera I ;

[0026] Analysis module for target-based C Target T L , and target T I With dynamic event graph T E The degree of contour overlap determines whether a dynamic target is found.

[0027] Furthermore, the analysis module includes:

[0028] Feature extraction module, used to extract the target T C Target T L and target T I The contour feature map is obtained by C , Feature map B L And feature map B I , and is also used to extract dynamic event graphs T E The event position features are obtained byE ;

[0029] Feature matching calculation module, used to calculate the feature map B C , Feature map B L , and feature map B I With feature map B E The degree of feature matching between them;

[0030] The judgment module is used to determine whether a dynamic target is found based on the calculated feature matching degree.

[0031] Beneficial Effects

[0032] Due to the adoption of the above-mentioned technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention constructs a multi-camera device including a color camera, a low-light camera, an infrared camera, and an event camera, and obtains the contour features of the target discovered by the color camera, the low-light camera, and the infrared camera through target detection and feature extraction, and then determines whether a dynamic target appears in the field of view by analyzing the degree of overlap between the contour features and the event position features collected by the event camera. Since the present invention combines the respective characteristics of different types of image data collected by the camera, it can effectively distinguish between the target and the background, adapt to the all-weather working environment, and is not easily deceived by false targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a structural diagram of a multi-camera device according to an embodiment of the present invention;

[0034] Figure 2 is a flow chart of a first embodiment of the present invention;

[0035] Figure 3 is a target found in a picture captured by a color camera according to an embodiment of the present invention and its corresponding contour feature map;

[0036] Figure 4 is a target found in a picture captured by the low-light camera according to an embodiment of the present invention and its corresponding contour feature map;

[0037] Figure 5 It is a target found in a picture collected by the infrared camera of an embodiment of the present invention and its corresponding contour feature map;

[0038] Figure 6 It is a dynamic event graph and its corresponding contour feature graph collected by the event camera of the embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0040] The first embodiment of the present invention relates to a dynamic target detection method based on a multi-camera device, which is applied to Figure 1 The multi-camera device shown includes a color camera, a low-light camera, an infrared camera, and an event camera. The specific method is as follows Figure 2 As shown, the following steps are included:

[0041] Get the images taken by the color camera, low-light camera, and infrared camera at the set time and perform target detection respectively to obtain the target T found by the color camera C , target T detected by low-light camera L , and the target T detected by the infrared camera I ;

[0042] Get the dynamic event graph T within the field of view discovered by the event camera at the set time E ;

[0043] Based on the target T C Target T L , and target T I With dynamic event graph T E The degree of contour overlap determines whether a dynamic target is found.

[0044] Among them, based on the target T C Target T L , and target T I With dynamic event graph T E The contour overlap determines whether a dynamic target is found, which can be achieved by the following methods:

[0045] Extract target T respectively C Target T L and target T I The contour feature map is obtained by C , Feature map B L And feature map B I ; Extract dynamic event graph T E The event position features are obtained by E ;

[0046] Calculate the feature map B separately C , Feature map B L , and feature map B I With feature map B EThe degree of feature matching between them;

[0047] If the calculated matching degree of any feature is greater than the set threshold, it is considered that a dynamic target is found.

[0048] The feature matching can be calculated by:

[0049] The feature map B C , Feature map B L , and feature map B I The pixel value of the contour area is set to the first set value, and the pixel value of the remaining area is set to the second set value, and the feature map B' is obtained. C , feature map B' L , and feature map B' I ;

[0050] The feature map B' C , feature map B' L , and feature map B' I As a Mask, in the feature map B E Move up, count the number of pixels in the overlapped area whose pixel values ​​are all the first set value, and divide it by the feature map B E After the total number of pixels is calculated, normalization is performed to obtain the feature matching degree.

[0051] In this implementation manner, the first setting value is set to 255 and the second setting value is set to 0, but they may also be set to other values.

[0052] Color cameras, low-light cameras, infrared cameras, and event cameras are not required to be installed on the same device, but are required to have overlapping fields of view.

[0053] Image processing for color cameras, low-light cameras, infrared cameras, and event cameras can be performed on the same computing platform or on different platforms, and the processing results can be finally gathered on one platform for dynamic target determination.

[0054] The color camera, low-light camera, and infrared camera do not need to detect the target at the same time. Subsequent judgment can be made as long as any one of the cameras detects the target.

[0055] The present embodiment is further described below in conjunction with the accompanying drawings and specific test results.

[0056] like Figure 1 The device shown includes a color camera C C , low light camera C L , infrared camera C I and event camera C E This arrangement ensures that all cameras have overlapping viewing angles.

[0057] First, four cameras are connected to the same computing unit, and images from the four cameras are acquired respectively.

[0058] Through the target detection algorithm, the color camera C C The target found in C ,like Figure 3 Left: In low-light camera C L The target found in L ,like Figure 4 As shown on the left; in the infrared camera C I The target found in I ,like Figure 5 Shown on the left;.

[0059] Use event cameras to detect dynamic events within the field of view T E ,like Figure 6 Shown on the left.

[0060] When the color camera, low-light camera, and infrared camera detect the target, the image is binarized and the contour features B of the target are calculated using the corrosion and expansion algorithms. C , B L , B I . Among them B C is the contour feature of the target in the color camera, such as Figure 3 As shown on the right; B L is the contour feature of the target in the low-light camera, such as Figure 4 As shown on the right; B I is the contour feature of the target in the infrared camera, such as Figure 5 Shown right.

[0061] Binarize the effective pixels in the dynamic camera (i.e., set the pixel value of the event position of the dynamic event map to 255, and the pixel value of the rest of the area to 0), and get the dynamic event T E The contour feature B E ,like Figure 6 Shown right.

[0062] Through feature matching, we get B C , B L , B I With B E The matching degree M CE , M LE , M IE The feature matching method is as follows: the pixel value of the background area of ​​the contour image is set to 0, and the pixel value of the contour area is set to 255; then the binarized B C , B L , B I As a Mask, in B EMove upward, count the number of pixels in the overlapping area whose pixel values ​​are all 255, and divide it by B E The total number of pixels is normalized to obtain the final matching degree M.

[0063] When M CE , M LE , M IE Any matching degree in is greater than the set matching threshold M Thre After that, the dynamic target is determined to be found.

[0064] A second embodiment of the present invention relates to a dynamic target detection system based on a multi-camera device, comprising:

[0065] Multi-camera setup, including color camera, low-light camera, infrared camera, and event camera;

[0066] The acquisition module is used to obtain the images taken by the color camera, the low-light camera, and the infrared camera at the set time, as well as the dynamic event map within the field of view discovered by the event camera at the set time. E ;

[0067] The target detection module is used to perform target detection on the images taken by the color camera, the low-light camera, and the infrared camera, and obtain the target T found by the color camera. C , target T detected by low-light camera L , and the target T detected by the infrared camera I ;

[0068] Analysis module for target-based C Target T L , and target T I With dynamic event graph T E The degree of contour overlap determines whether a dynamic target is found.

[0069] The analysis module includes:

[0070] Feature extraction module, used to extract the target T C Target T L and target T I The contour feature map is obtained by C , Feature map B L And feature map B I , and is also used to extract dynamic event graphs T E The event position features are obtained by E ;

[0071] Feature matching calculation module, used to calculate the feature map B C , Feature map B L , and feature map B IWith feature map B E The degree of feature matching between them;

[0072] The judgment module is used to determine whether a dynamic target is found based on the calculated feature matching degree. If any calculated feature matching degree is greater than a set threshold, it is determined that a dynamic target is found.

[0073] Color cameras, low-light cameras, infrared cameras, and event cameras are not required to be installed on the same device, but are required to have overlapping fields of view.

[0074] Image processing for color cameras, low-light cameras, infrared cameras, and event cameras can be performed on the same computing platform or on different platforms, and the processing results can be finally gathered on one platform for dynamic target determination.

[0075] The color camera, low-light camera, and infrared camera do not need to detect the target at the same time. Subsequent judgment can be made as long as any one of the cameras detects the target.

[0076] The present embodiment is further described below in conjunction with the accompanying drawings and specific test results.

[0077] like Figure 1 The device shown includes a color camera C C , low light camera C L , infrared camera C I and event camera C E This arrangement ensures that all cameras have overlapping viewing angles.

[0078] First, four cameras are connected to the same computing unit, and images from the four cameras are acquired respectively.

[0079] Through the target detection algorithm, the color camera C C The target found in C ,like Figure 3 Left: In low-light camera C L The target found in L ,like Figure 4 As shown on the left; in the infrared camera C I The target found in I ,like Figure 5 Shown on the left;.

[0080] Use event cameras to detect dynamic events within the field of view T E ,like Figure 6 Shown on the left.

[0081] When the color camera, low-light camera, and infrared camera detect the target, the image is binarized and the contour features B of the target are calculated using the corrosion and expansion algorithms. C , BL , B I . Among them B C is the contour feature of the target in the color camera, such as Figure 3 As shown on the right; B L is the contour feature of the target in the low-light camera, such as Figure 4 As shown on the right; B I is the contour feature of the target in the infrared camera, such as Figure 5 Shown right.

[0082] Binarize the effective pixels in the dynamic camera (i.e., set the pixel value of the event position of the dynamic event map to 255, and the pixel value of the rest of the area to 0), and get the dynamic event T E The contour feature B E ,like Figure 6 Shown right.

[0083] Through feature matching, we get B C , B L , B I With B E The matching degree M CE , M LE , M IE The feature matching method is as follows: the pixel value of the background area of ​​the contour image is set to 0, and the pixel value of the contour area is set to 255; then the binarized B C , B L , B I As a Mask, in B E Move upward, count the number of pixels in the overlapping area whose pixel values ​​are all 255, and divide it by B E The total number of pixels is normalized to obtain the final matching degree M.

[0084] When M CE , M LE , M IE Any matching degree in is greater than the set matching threshold M Thre After that, the dynamic target is determined to be found.

Claims

1. A dynamic target detection method based on a multi-camera device, characterized in that: The invention is applied to a multi-camera device including a color camera, a low-light camera, an infrared camera, and an event camera, and comprises the following steps: Get the images taken by the color camera, low-light camera, and infrared camera at the set time and perform target detection respectively to obtain the target T found by the color camera C , target T detected by low-light camera L , and the target T detected by the infrared camera I ; Get the dynamic event graph T within the field of view discovered by the event camera at the set time E ; Based on the target T C Target T L , and target T I With dynamic event graph T E The degree of contour overlap determines whether a dynamic target is found.

2. The method according to claim 1, characterized in that The target T C Target T L , and target T I With dynamic event graph T E The degree of contour overlap determines whether a dynamic target is found, including: Extract target T respectively C Target T L and target T I The contour feature map is obtained by C , Feature map B L And feature map B I ; Extract dynamic event graph T E The event position features are obtained by E ; Calculate the feature map B separately C , Feature map B L , and feature map B I With feature map B E The degree of feature matching between them; Whether a dynamic target is found is determined based on the calculated feature matching degree.

3. The method according to claim 3, characterized in that: The feature matching degree is calculated by the following method: The feature map B C , Feature map B L , and feature map B I The pixel value of the contour area is set to the first set value, and the pixel value of the remaining area is set to the second set value, and the feature map B′ is obtained. C , feature map B′ L , and feature map B′ I ; The feature graph B′ C , feature map B′ L , and feature map B′ I As a Mask, in the feature map B E Move up, count the number of pixels in the overlapped area whose pixel values ​​are all the first set value, and divide it by the feature map B E After the total number of pixels is calculated, normalization is performed to obtain the feature matching degree.

4. The method according to claim 3, characterized in that The extracted dynamic event graph T E The event position features are obtained by E ,include: The dynamic event graph T E The pixel value of the event position is set to the first set value, and the pixel values ​​of the remaining areas are set to the second set value, and the feature map B is obtained. E .

5. The method according to claim 4, characterized in that The first setting value is 255, and the second setting value is 0.

6. The method according to claim 2, characterized in that The determining whether a dynamic target is found according to the calculated feature matching degree includes: If the calculated matching degree of any feature is greater than the set threshold, it is considered that a dynamic target is found.

7. The method according to claim 1, characterized in that The color camera, low-light camera, infrared camera, and event camera have overlapping fields of view.

8. A dynamic target detection system based on a multi-camera device, characterized in that: include: Multi-camera setup, including color camera, low-light camera, infrared camera, and event camera; The acquisition module is used to obtain the images taken by the color camera, the low-light camera, and the infrared camera at the set time, as well as the dynamic event map within the field of view discovered by the event camera at the set time. E ; The target detection module is used to perform target detection on the images taken by the color camera, the low-light camera, and the infrared camera, and obtain the target T found by the color camera. C , target T detected by low-light camera L , and the target T detected by the infrared camera I ; Analysis module for target-based C Target T L , and target T I With dynamic event graph T E The degree of contour overlap determines whether a dynamic target is found.

9. The system according to claim 8, characterized in that The analysis module comprises: Feature extraction module, used to extract the target T C Target T L and target T I The contour feature map is obtained by C , Feature map B L And feature map B I , and is also used to extract dynamic event graphs T E The event position features are obtained by E ; Feature matching calculation module, used to calculate the feature map B C , Feature map B L , and feature map B I With feature map B E The degree of feature matching between them; The judgment module is used to determine whether a dynamic target is found based on the calculated feature matching degree.

Images