A planar bionic compound eye imaging device and a method for extracting dynamic vision features

Abstract

The present invention discloses a planar bionic compound eye imaging device and a dynamic vision feature extraction method, belonging to the field of image dynamic feature extraction and recognition. The device includes seven identical image acquisition devices, one located at the center, and the remaining six arranged in a regular hexagon structure around the central device. The sub-eye images generated by adjacent devices have a 60% - 80% overlap, making full use of the hexagonal structure and imaging characteristics of the compound eye, laying a foundation for calculating the optical flow through the image differences between adjacent cameras. The method divides the dynamic vision features into translational features and approaching features, uses the bionic optical flow method to analyze the imaging differences of the target motion in multiple adjacent sub-eye images, and extracts the translational optical flow generated by the target motion in the central sub-eye image; uses the bionic motion detector to analyze the imaging of the target motion in the central sub-eye image, extracts the approaching optical flow generated by the target motion in the central sub-eye image, and normalizes the two types of optical flows as the dynamic vision features of the target image.

Description

Technical Field

[0001] The present invention belongs to the field of image dynamic feature extraction and recognition, and more specifically, relates to a planar bionic compound eye imaging device and a dynamic vision feature extraction method. Background Art

[0002] Traditional vision feature extraction technologies usually take the extraction of static features (size, shape, texture, etc.) of target images as the main object, and have weak ability to extract dynamic features (trajectory, speed, and direction) of target images in complex dynamic environments. Although small target extraction technologies can extract dynamic features from target images, their efficiency is not high in complex environments. Therefore, there is an urgent need for vision feature extraction devices and methods that are more suitable for high-complexity and strong-dynamic environments. The compound eye structure of insects is more adept at perceiving the dynamic vision features of targets through optical flow measurement, and is insensitive to static vision features. This feature extraction mechanism mainly based on dynamic vision enables insects to have the ability to detect targets in high-complexity and strong-dynamic environments that surpass higher animals under extremely harsh physiological conditions. Therefore, by drawing on the compound eye structure and dynamic vision mechanism of insects, it is expected to construct an efficient method and device for extracting dynamic vision features in complex environments from the bionic perspective of optical flow measurement, thereby providing a way for the detection of high-speed and small targets.

[0003] Patent CN100373394C simulates the compound eye structure of insects by splicing 7 cameras, and the 7 cameras used are arranged in an arc surface. This patent does not use the optical flow measurement method to extract the dynamic features of the target image, but instead, according to the image differences collected by the cameras at different times, uses the image segmentation method to remove the static background of the target from multiple frames of images, and then filters out the dynamic features of the target. If the device of this patent uses the optical flow method, it is necessary to calibrate the images arranged in an arc surface collected to generate images arranged in a plane, and the calibration process is prone to pixel position errors, thus causing errors in optical flow measurement. Summary of the Invention

[0004] Aiming at the defects of the prior art, the purpose of the present invention is to provide a planar bionic compound eye imaging device and a dynamic vision feature extraction method, aiming to use the optical flow measurement method to extract the dynamic features of the target image and avoid the measurement errors introduced by the calibration process.

[0005] To achieve the above object, in the first aspect, the present invention provides a planar bionic compound eye imaging device, and the device includes: 7 identical image acquisition devices and a control module;

[0006] Among the seven identical image acquisition devices, one is located at the center, and the remaining six are arranged in a regular hexagon structure around the central image acquisition device. The lenses of all image acquisition devices are on the same plane, with equal spacing and the same orientation, and the sub-eye images generated by adjacent two image acquisition devices have a 60% - 80% overlap;

[0007] The control module is used to control the synchronous acquisition of the seven image acquisition devices.

[0008] Preferably, the sub-eye image is a regular hexagon image intercepted from the central area of the image acquired by the image acquisition device, and the seven sub-eye images generated by the seven image acquisition devices have the same size.

[0009] Preferably, the device further includes: an adjustment module, which is used to synchronously adjust the spacing and / or orientation of all image acquisition devices to ensure that the lens spacings of adjacent image acquisition devices are equal.

[0010] To achieve the above object, in a second aspect, the present invention provides a method for extracting dynamic visual features based on a bionic compound eye, and the method includes:

[0011] S1. Using the planar bionic compound eye imaging device as described in the first aspect, perform visual image acquisition on the target scene for which dynamic visual features need to be extracted to obtain seven sub-eye images;

[0012] S2. According to the differences between the central sub-eye image and adjacent sub-eye images at the same moment, use the image interpolation method to respectively extract the translational optical flow of each pixel point in the central sub-eye image; according to the differences between the central sub-eye images at adjacent moments, use the lobula giant movement detector model to respectively extract the approaching optical flow of each pixel point in the central sub-eye image;

[0013] S3. Normalize the translational optical flow and the approaching optical flow of each pixel point respectively, and use them as the dynamic visual features extracted by the planar bionic compound eye imaging device.

[0014] Preferably, before performing optical flow calculation, use a Gaussian low-pass filter to perform low-pass filtering on the sub-eye images respectively.

[0015] Preferably, the specific method for extracting the translational optical flow of each pixel point in the central sub-eye image is as follows:

[0016] (1) Obtain the central sub-eye image f acquired at time t and six other sub-eye images f i± , i = x, y, z. The central sub-eye image saved at the previous moment t - 1 is f0, and the area A is a regular hexagon area centered at the point (, y) in each sub-eye image;

[0017] (2) Use the following system of three linear equations to calculate the translational optical flow OF(x,y) of the central sub-eye image at time t at the pixel point (x,y) = x ,y,z]:

[0018]

[0019] where T x ,y,z respectively represent the displacement amounts of the pixel point (x,y) along the three coordinate axes when the central sub-eye image changes from f0 to f, and the Gaussian window function r represents the pixel distance between the center points of two adjacent sub-eye images, and q represents the full width at half maximum of the Gaussian window function.

[0020] Preferably, the approximation optical flow of each pixel point in the central sub-eye image is extracted as follows:

[0021] (1) Obtain the central sub-eye image f collected at time t, the central sub-eye image f0 saved at the previous time t-1, and the central sub-eye image f saved at time t-2 -1 , and region A is a regular hexagon region in the central sub-eye image centered on the point (x,y). Region A is divided clockwise into 6 sub-regions A i , i = 1,2,...,6;

[0022] (2) Calculate the absolute value of the difference between each pixel point in images f and f0:

[0023] F(x,y) = |f(x,y) - f0(x,y)|

[0024] (3) Calculate the local suppression value of each pixel point (x,y) in region A at time t-1:

[0025] [[ID=?]] [[ID=?]] [[ID=?]]

[0026] where F0(x,y) = |f0(x,y) - f -1 (x,y)|;

[0027] (4) Calculate the membrane potential in region A at time t:

[0028] S(x,y) = F(x,y) - I(x,y)W, (x,y) ∈ A

[0029] (5) Low-pass filter the membrane potential in region A:

[0030]

[0031] (6) Set the threshold T G for filtering: It should be noted that there are some question marks in the translation where the original text seems to be incomplete or unclear in those parts. You may need to check and correct the original text for a more accurate translation.

[0032]

[0033] (7) Calculate the approximate optical flow IF_N(x, y) of the central sub-eye image at pixel point (x, y) at time t = [N x , N y , N z :

[0034]

[0035]

[0036]

[0037] where F0(x, y) represents the absolute value of the pixel point difference calculated at the previous time t - 1, ω(i, j) represents the coefficient matrix of the local suppression weight, W represents the suppression weight, and ω e represents the convolution kernel with spatial low-pass filtering, represents the filtered membrane potential, C represents the attenuation coefficient in [0, 1], and N x , N y , N z respectively represent the approximate displacement amounts generated at pixel point (x, y) along the three coordinate axes when the central sub-eye image changes from f0 to f.

[0038] Preferably, use the Sigmoid function to normalize the three translational optical flow amounts OF_T(x, y) = [T x , T y , T z to obtain the translational feature at pixel point (x, y) on the central image T x , T y , T z respectively represent the translational displacement amounts generated at pixel point (x, y) along the three coordinate axes when the central sub-eye image changes from f0 to f.

[0039] Preferably, use the Sigmoid function to normalize the three approximate optical flows OF_N(, y) = [N x , N y , N z to obtain the approximate feature at pixel point (x, y) on the central image N x , N y , N z respectively represent the approximate displacement amounts generated at pixel point (x, y) along the three coordinate axes when the central sub-eye image changes from f0 to f, and M is the normalization adjustment coefficient.

[0040] For achieving the above object, in a third aspect, the present invention provides a computer-readable storage medium storing a computer program, which when executed by a processor can implement the method described in the second aspect.

[0041] Generally speaking, compared with the prior art by the above technical solution conceived by the present invention, the following beneficial effects are obtained:

[0042] (1) Different from the existing curved surface bionic compound eye imaging device, the present invention proposes a planar bionic compound eye imaging device, which includes 7 identical image acquisition devices, with 1 located in the center and the remaining 6 arranged in a regular hexagon structure around the central image acquisition device. The lenses of all image acquisition devices are located on the same plane, with equal spacing and the same orientation, and the sub-eye images generated by adjacent two image acquisition devices have a 60% - 80% overlap, making full use of the hexagonal layout structure and imaging characteristics of adjacent sub-eyes in the compound eye, laying a foundation for calculating image optical flow through the image differences of adjacent cameras.

[0043] (2) The present invention proposes a method for extracting dynamic visual features based on a bionic compound eye, which divides dynamic visual features into two parts: translation features and approaching features. The bionic optical flow calculation method is used to analyze the imaging differences of a target in multiple adjacent sub-eye images during movement, and the translation optical flow generated by the target movement in the central sub-eye image is extracted; the bionic motion detector is used to analyze the imaging of the target in the central sub-eye image during movement, and the approaching optical flow generated by the target movement in the central sub-eye image is extracted. The extracted translation and approaching optical flows are normalized and used as the dynamic visual features of the target image, providing a solution for the extraction of dynamic features of high-complexity, strong-dynamic, and weak-imaging targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural diagram of a planar bionic compound eye imaging device provided by the present invention.

[0045] Figure 2 It is a flowchart of a method for extracting dynamic visual features based on a bionic compound eye provided by the present invention.

[0046] Figure 3 It is a sub-flowchart of step S1 provided by the present invention.

[0047] FIG. 4(a) is a structural diagram of a sub-eye image provided by the present invention.

[0048] FIG. 4(b) is an imaging superposition diagram of a sub-eye image provided by the present invention.

[0049] Figure 5 It is a translation optical flow diagram of a certain pixel point in the central sub-eye image provided by the present invention.

[0050] Figure 6The approximated optical flow map of a certain pixel in the central sub-eye image provided by the present invention.

[0051] Figure 7 The sub-flowchart of step S3 provided by the present invention. Detailed implementation manners

[0052] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] As Figure 1 shown, the present invention provides a planar bionic compound eye imaging device, which includes seven identical cameras, where one camera is a central camera, and the remaining six cameras are arranged in a regular hexagon structure and evenly distributed around the central camera. The lenses of the seven cameras are located on the same plane, with equal spacing and the same orientation. The spacing between each camera is r. The camera spacing r can be dynamically adjusted according to usage requirements, but all seven cameras must be adjusted simultaneously. The image collected by the central camera is called the central sub-eye image.

[0054] As Figure 2 shown, the present invention provides a dynamic vision feature extraction method based on a bionic compound eye, which specifically includes the following steps:

[0055] S1. Use the device of the present invention to collect seven sub-eye images.

[0056] As Figure 3 shown, the step S1 includes the following sub-steps:

[0057] S11. Use the vision feature extraction device of the present invention to simultaneously collect seven images by seven cameras.

[0058] S12. Respectively intercept the regular hexagon regions in the centers of the seven images to generate seven sub-eye images, where the sub-eye image generated by the central camera is the central sub-eye image.

[0059] As shown in Figure 4(a), taking the generation of the central sub-eye image as an example, a regular hexagon image is extracted from the central region of the image collected by the central camera, and this sub-eye image is marked as f. The methods for generating sub-eye images by other cameras are the same. Among them, the sub-eye image generated by camera 1 is marked as f x+ , the sub-eye image generated by camera 2 is marked as f y- , the sub-eye image generated by camera 3 is marked as f z+ , the sub-eye image generated by camera 4 is marked as f x- , the sub-eye image of camera 5 is marked as f y+ , the sub-eye image of camera 6 is marked as f z- .

[0060] As shown in Figure 4(b), the content of the seven sub-eye images is not mutually independent. Instead, it is necessary to ensure that the sub-eye images generated by adjacent cameras have a 70% - 80% overlap. The adjustment of the overlap can be achieved by modifying Figure 1 the distance r between the cameras in. However, if the overlap is too large, the difference between the images is not obvious, and the extracted dynamic features are inaccurate. In this case, it is necessary to increase the distance r; if the overlap is too small, the correlation between the images is less, and the extracted dynamic features are also inaccurate. In this case, it is necessary to reduce the distance r.

[0061] S13. Perform low-pass filtering on the seven sub-eye images respectively.

[0062] When the high-frequency components in the image are large, the translational optical flow calculation method used in the present invention will produce large errors. The present invention preferably uses a spatially smoothed Gaussian low-pass filter to perform a convolution operation on the seven sub-eye images to remove the high-frequency components in the sub-eye images and make the optical flow calculation more accurate.

[0063] S2. Extract the translational optical flow of a certain pixel point in the filtered central sub-eye image.

[0064] The extraction of the translational optical flow uses an image interpolation method. According to the spatial correlation of the above seven sub-eye images, the translational optical flow of the central sub-eye is estimated from the change in the central sub-eye image and the differences in other sub-eye images.

[0065] Taking a pixel point with coordinates (,y) on the central sub-eye image as an example, the process of extracting the translational optical flow of this point is described. As shown in Figure 4(a), assume that the device of the present invention acquires the central sub-eye image f and six other sub-eye images f i± (=x,y,z) at time t. The central sub-eye image saved at the previous time t - 1 is f0, and region A is a hexagonal region centered at the point (,y) in each sub-eye image. As Figure 5 shown, assume that when the central sub-eye image changes from time t - 1 to t, from image f0 to f, the pixel point (,y) generates translational displacement amounts T x , T y , T z . Thus, the present invention defines the translational optical flow of the central sub-eye image at pixel point (,y) at time t as OF_(,y) = x ,y,z], and uses the following system of three linear equations to calculate OF_(,y):

[0066]

[0067] Among them, the Gaussian window function Let r denote the pixel distance between the centers of two adjacent sub - eye images, and q denote the full width at half maximum of the Gaussian window function. Solving this system of equations can obtain the three translational optical flows T x 、T y 、T z at the pixel point (x, y) of the central sub - eye image.

[0068] The above formula calculates the optical flow at the center point of region A. Pixels closer to the center point contribute more to the optical flow calculation, while pixels farther from the center point contribute less. Therefore, the function is a Gaussian window function with the same size and shape as region A. The function value is larger when closer to the center point and smaller when farther from the center point. In the above formula, the pixel value at each point (x, y) in region A is multiplied by the weight value calculated by this function using (x, y), thereby highlighting the importance of pixels closer to the center of region A in the optical flow calculation. The distance between the centers of every two adjacent sub - eye images is r. q is the full width at half maximum of the Gaussian window function. The smaller the q value, the higher the importance of pixels closer to the center of region A in the optical flow calculation.

[0069] Similarly, the translational optical flow of each pixel point on the central sub - eye image can be obtained by the above method.

[0070] S3. Extract the approximate optical flow of a certain pixel point on the central sub - eye image.

[0071] Specifically, the extraction of the approximate optical flow uses the lobular giant motion detector model to estimate the approximate optical flow of the central sub - eye based on the change between the central sub - eye image at the current moment and the central sub - eye image at the previous moment.

[0072] As Figure 6 shown, taking a pixel point with coordinates (x, y) on the central sub - eye image as an example, the process of extracting the approximate optical flow of this point is described. Assume that the device of the present invention acquires the central sub - eye image f at time t, saves the central sub - eye image f0 at the previous time t - 1, and saves the central sub - eye image f -1 at time t - 2. Region A is a hexagonal region centered at the point (x, y), and region A is divided into 6 sub - regions A i (i = 1, 2, …, 6) with equal areas. When the central sub - eye image changes from f0 to f, the point (x, y) generates approximate displacement amounts N x 、N y 、N z along the three coordinate axes respectively. Thus, the present invention defines the approximate optical flow of the central sub - eye image at the pixel point (x, y) as OF_N(x, y)=[N x ,N y ,N z .

[0073] like Figure 7 As shown, when calculating the approximate optical flow at time t, step S3 includes the following sub-steps:

[0074] S31, calculate the absolute value of the difference between each pixel in images f and f0:

[0075] F(x,y)=|f(x,y)-f0(x,y)|

[0076] S32. Calculate the local suppression value of each pixel (x, y) in region A at the previous moment:

[0077]

[0078] F0(x,y)=|f0(x,y)-f -1 (x,y)|

[0079] Wherein, F0(x,y) is the absolute value of the pixel difference value obtained by executing step S31 at the previous moment, ω(i,j) is the coefficient matrix of the local suppression weight, and optionally,

[0080]

[0081] S33. Calculate the membrane potential in region A:

[0082] S(x,y)=F(x,y)-I(x,y)W,(x,y)∈A

[0083] Where W is the suppression weight.

[0084] S34. Low-pass filter the membrane potential in region A:

[0085]

[0086] Among them, ω e is a convolution kernel with spatial low-pass filtering, optionally,

[0087]

[0088] S35. Setting threshold T G , filter the point G with smaller value, that is:

[0089]

[0090] in, is the membrane potential after filtering, and C is the attenuation coefficient in [0,1].

[0091] S36. Calculate the approximate optical flow OF_N(x,y) of the central sub-eye image at the pixel point (x,y) at time t.

[0092] According to Figure 6 the six image regions A i (i = 1, 2, …, 6), after synthesizing the in each region as the overall membrane potential generated by this region, then add the overall membrane potentials corresponding to two regions as the optical flow in this direction. Therefore, the calculation of OF_N(x, y) = [N x , N y , N z is as follows:

[0093]

[0094]

[0095]

[0096] Similarly, the approximate optical flow of each pixel on the central ommatidium image can be obtained by the above method.

[0097] S4. Normalize the extracted translational optical flow as the translational feature of this pixel.

[0098] Since different calculation methods are used for translational optical flow and approximate optical flow, and they have different value ranges, it is necessary to normalize the three translational optical flow quantities and the three approximate optical flow quantities to ensure that the six dynamic vision features at this pixel in the image have a unified measurement standard, which is convenient for subsequent feature recognition and analysis.

[0099] Normalize the three translational optical flow quantities OF_T(x, y) = [T x , T y , T z to obtain the translational feature Feature_T(x, y) at the pixel point (x, y) on the central image. Optionally, the mechanism of normalizing the input signal of bionic insect neurons uses the Sigmoid function for normalization:

[0100]

[0101] S5. Normalize the extracted approximate optical flow as the approximate feature of this pixel.

[0102] Normalize the three approximate optical flow quantities OF_N(x, y) = [N x , N y , N z to obtain the approximate feature Feature_N(x, y) at the pixel point (x, y) on the central image:

[0103]

[0104] Among them, M is a normalization adjustment coefficient. Preferably, M = the number of pixel points in region A / 3.

[0105] S6. Determine whether there are pixel points with features to be extracted on the central sub-eye image. If so, extract the positions of these pixel points, and then go to step S2; if not, go to step S7.

[0106] Specifically, Feature_T(x, y) and Feature_N(x, y) are the dynamic vision features extracted from the central sub-eye image (x, y) of the bionic compound eye in the present invention. Similarly, the dynamic vision features of other pixel points in the central sub-eye image can all be extracted by the above method.

[0107] S7. Save the currently acquired central sub-eye image.

[0108] Specifically, save the image f and label it as f0, and save all F(x, y) calculated in step S31 and label it as F0(, y).

[0109] S&. Determine whether the image acquisition device is shut down. If so, exit; if not, go to step S1.

[0110] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present invention, and are not used to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should all be included within the protection scope of the present invention.

Claims

1. A method for extracting dynamic visual features based on a bionic compound eye, characterized in that The method includes: S1. Using a planar bionic compound eye imaging device to collect visual images of a target scene for which dynamic visual features need to be extracted, and obtaining 7 sub-eye images; S2. According to the differences between the central sub-eye image and adjacent sub-eye images at the same moment, using an image interpolation method to separately extract the translational optical flow of each pixel point in the central sub-eye image; according to the differences between the central sub-eye images at adjacent moments, using a lobula giant movement detector model to separately extract the approaching optical flow of each pixel point in the central sub-eye image; S3. Separately performing normalization processing on the translational optical flow and the approaching optical flow of each pixel point, which are used as the dynamic visual features extracted by the planar bionic compound eye imaging device; The planar bionic compound eye imaging device includes: 7 identical image acquisition devices and a control module; among the 7 identical image acquisition devices, 1 is located at the center, and the remaining 6 are arranged in a regular hexagon structure around the central image acquisition device. The lenses of all image acquisition devices are located on the same plane, with equal spacing, the same orientation, and the sub-eye images generated by adjacent two image acquisition devices have a 60% - 80% overlap; the control module is used to control the synchronous acquisition of the 7 image acquisition devices; The specific steps of separately extracting the translational optical flow of each pixel point in the central sub-eye image by using the image interpolation method are as follows: (1) Obtain at the moment The central sub-eye image collected And 6 other sub-eye images , , at the previous moment The saved central sub-eye image is , area A is a regular hexagon area centered on the point in each sub-eye image; (2) Use the following system of three linear equations to calculate the moment The central sub-eye image at the pixel point of the translational optical flow : Among them, respectively represent the displacement amounts of translation generated by the pixel point when changing from along the three coordinate axes, the Gaussian window function , represents the pixel distance between the central points of two adjacent sub-eye images, represents the full width at half maximum of the Gaussian window function; The specific steps of extracting the approaching optical flow of each pixel point in the central sub-eye image are as follows: (1) Obtain at the moment The central sub-eye image collected , and at the previous moment The central sub-eye image saved , and at the moment The central sub-eye image saved , where Region A is a regular hexagon region centered at point in the central sub-eye image, and Region A is divided clockwise into 6 sub-regions with equal areas , ; (2)Calculate the image and the absolute value of the difference between each pixel point in: (3) Calculate the local suppression values of each pixel point within the time zone A: Among them, ; (4) Calculate The membrane potential within the time zone A: (5) Performing low-pass filtering on the membrane potential in region A; (6) Set a threshold value Perform filtering: (7) Calculation time The central sub-eye image at pixel point Approximate optical flow : Among them, represents the absolute value of the pixel difference calculated at the previous moment ; represents the coefficient matrix of the local suppression weight represents the suppression weight represents the convolution kernel with spatial low-pass filtering represents the membrane potential after filtering represents the attenuation coefficient in [0, 1] respectively represent that when the central sub-eye image changes from to at time point the approximate displacement amounts generated along the three coordinate axes respectively.

2. The method according to claim 1, wherein The sub-eye image is a regular hexagon image intercepted from the central region of the image collected by the image acquisition device, and the 7 sub-eye images generated by the 7 image acquisition devices have the same size.

3. The method according to claim 1, wherein The device further includes: an adjustment module for synchronously adjusting the spacing and / or orientation of all image acquisition devices to ensure that the lens spacing of adjacent image acquisition devices is equal.

4. The method according to claim 1, wherein Before performing optical flow calculation, use a Gaussian low-pass filter to perform low-pass filtering on the sub-eye images respectively.

5. The method according to claim 1, wherein The Sigmoid function is used to normalize the three translational optical flows to obtain the translational features at the pixel points on the central image . respectively represent the displacement amounts of the translations generated along the three coordinate axes when the central sub-eye image changes from to at the pixel point.

6. The method according to claim 1, wherein The three approximate optical flows are normalized using the Sigmoid function to obtain the approximate features at the pixel points on the central image ; among which respectively represent the approximate displacement amounts generated by the point along the three coordinate axes when the central sub-eye image changes from to ; is the normalization adjustment coefficient. ​​ 7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement the method according to any one of claims 1 to 6.

Images