An active localization method of biomimetic eye binocular vision for performing slam task

Abstract

The application belongs to the technical field of robot vision, and discloses an active positioning method of bionic eye binocular vision for performing a SLAM task, which comprises the following steps: firstly, a panoramic camera is used to acquire a panoramic image, and key scene information is detected; then, pixel points in the panoramic image are valued based on the key scene information, and a panoramic value image is obtained; then, the field of view regions of left and right bionic eye cameras are projected into the panoramic value image respectively, and a current binocular field of view projection region is obtained; the value average of the current binocular field of view projection region is obtained in the panoramic value image; the value average of the current binocular field of view projection region is compared with a value threshold, and a binocular field of view projection region greater than the value threshold is obtained by moving the binocular bionic eye camera; finally, high value images collected by the left and right bionic eye cameras are used as inputs of a SLAM system. The method has important practical application value, and can be widely applied in the fields of robot navigation, automatic driving and the like.

Description

Technical Field

[0001] This invention belongs to the field of robot vision technology, specifically relating to an active localization method using bionic binocular vision for performing SLAM tasks. Background Technology

[0002] The eye is one of the most important organs for humans to interact with the external environment; humans obtain over 90% of their perception of the external environment through their eyes. For bionic robots, the importance of eyes in correctly performing complex tasks is self-evident. With the continuous development of robot vision technology, users' demands for the ability of bionic eye robots to acquire external information are gradually increasing, requiring bionic eyes to actively acquire key information from the visual information of the entire scene. However, most current robot vision systems simply fix visual sensors directly to the robot, and their visual perception capabilities are insurmountably different from those of the human eye. For example, the human eye can achieve rapid and accurate gaze localization, smoothly track moving targets, and maintain gaze stability during head movements through the vestibular-ocular reflex mechanism, but bionic eyes cannot achieve these.

[0003] Human vision is divided into central vision and peripheral vision. Central vision plays the most important role in recognizing specific objects, while peripheral vision is more crucial for perceiving the overall scene. When robots equipped with visual sensors perform SLAM in unknown environments, the system primarily passively receives image data collected by the visual sensors during robot movement, using this data to perceive the robot's pose and surrounding environment. When the images collected by the visual sensors contain sparse textures or numerous dynamic objects, the SLAM system faces tracking failures. Therefore, inspired by the human eye's peripheral and central vision perception mechanisms, a bionic eye system with both peripheral and central vision can be constructed, enabling the system to possess active visual capabilities. During SLAM tasks, the bionic eye can actively acquire key information from the visual information of the entire scene and control movement to shift the gaze to areas of critical information (high value). Summary of the Invention

[0004] In view of the problems and shortcomings of the existing technology, the purpose of this invention is to provide an active localization method for bionic binocular vision when performing SLAM tasks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The first aspect of this invention provides an active localization method using bionic binocular vision for performing SLAM tasks, comprising the following steps:

[0007] The S100 uses a panoramic camera to acquire panoramic images, and then detects key scene information in the panoramic images that affects the execution of the SLAM task.

[0008] S200 assigns values ​​to pixels in the panoramic image based on key scene information to obtain a panoramic value image (on the pixel plane of the panoramic camera).

[0009] S300 projects the field of view areas of the left and right bionic eye cameras onto the panoramic value image respectively, to obtain the left field of view projection area and the right field of view projection area; at the same time, it controls the overlap of the left field of view projection area and the right field of view projection area; and merges the left field of view projection area and the right field of view projection area to obtain the current binocular field of view projection area.

[0010] S400, obtains the average value of the current binocular field-of-view projection area in the panoramic value image;

[0011] S500, determine the value mean of the current binocular field of view projection area and the value threshold: if the value mean of the current binocular field of view projection area is greater than the value threshold, do not move the current binocular field of view projection area, and use the high-value images currently acquired by the left and right bionic eye cameras as the input of the SLAM system; if the value mean of the current binocular field of view projection area is less than or equal to the value threshold, search for a target binocular field of view projection area in the panoramic value image whose value mean is higher than the value threshold, and then proceed to step S600;

[0012] S600: Based on the current binocular field of view projection area and the target binocular field of view projection area, calculate the required displacement of the left and right bionic eye cameras respectively; move the left and right bionic eye cameras respectively according to the displacement amount, and finally use the high-value images acquired by the left and right bionic eye cameras in the target binocular field of view projection area as the input of the SLAM system.

[0013] Preferably, the panoramic camera can capture 360° scene information. Furthermore, the purpose of using the panoramic camera is to simulate the peripheral vision of the human eye.

[0014] Preferably, in step S100, the key scene information is scene information that affects the running accuracy and robustness of the SLAM task; the key scene information includes feature points and dynamic objects.

[0015] Preferably, in step S100, the feature point extraction method is to extract the feature points using the FAST keypoint detection algorithm. During feature point extraction, since feature matching is not required, the calculation of the BRIEF descriptor in the ORB feature points extracted by the SLAM algorithm is omitted. Further, FAST is a corner detection algorithm, primarily used to detect areas with significant local pixel grayscale changes, and is known for its high speed. Its core idea is: if a pixel differs significantly from its neighboring pixels (too bright or too dark), it is more likely to be a corner point. The detection process of the FAST keypoint detection algorithm is as follows:

[0016] (1) Select pixel p in the image, and assume its brightness is I. p ;

[0017] (2) Set a threshold T(I) p 20% of;

[0018] (3) Select 16 pixels on a circle with a radius of 3, with pixel p as the center;

[0019] (4) If the selected circle has N consecutive points with a brightness greater than I p +T or less than I p - The pixel p in T can be considered a feature point (N is usually 12, i.e., FAST-12);

[0020] (5) Repeat the above 4 steps and perform the same operation on each pixel.

[0021] Preferably, in step S100, the method for detecting dynamic objects is to use the YOLO v7 detection algorithm to detect the dynamic objects; the dynamic objects are represented by detection bounding boxes. For dynamic object detection, the YOLO v7 detection algorithm is used to detect dynamic objects such as pedestrians and animals in the panoramic image, or potential dynamic objects that have a negative impact on map reuse.

[0022] Preferably, in step S200, the step of assigning values ​​to pixels in the panoramic image based on key scene information specifically involves: assigning high grayscale values ​​to pixels located at feature points and pixels located in the neighborhood of feature points in the panoramic image, and assigning low grayscale values ​​to all pixels located within the detection target box of dynamic objects; wherein, the grayscale value assigned to the pixels located in the neighborhood of feature points decreases as the neighborhood radius increases; if the same pixel is assigned multiple values, the pixel takes the maximum value assigned to it.

[0023] Furthermore, since feature points are the primary focus of the SLAM algorithm, feature points and their neighborhoods are assigned high grayscale values; the standard for high grayscale values ​​is a grayscale value higher than 240. For dynamic objects that severely degrade the performance of the SLAM algorithm, all pixels in the detection bounding box of the dynamic object are assigned low grayscale values, even lower than areas where no feature points were detected; the standard for low grayscale values ​​is a grayscale value lower than 80.

[0024] Preferably, in step S300, the step of projecting the field of view regions of the left and right bionic eye cameras onto the panoramic value image is specifically as follows: using the normalized plane projection equation of the bionic eye camera and the normalized spherical projection equation of the panoramic camera, the projection of the field of view regions (pixel planes) of the left and right bionic eye cameras onto the panoramic value image (pixel plane) of the panoramic camera is calculated respectively.

[0025] More preferably, the step of calculating the projection of the pixel planes of the left and right bionic eye cameras onto the pixel plane of the panoramic camera specifically includes:

[0026] S301, using the normalized plane projection equation of the bionic eye camera, calculates the four pixels u located at the vertices of the pixel plane of the left bionic eye camera. L(0,0) u L(m,0) u L(m,n) and u L(0,n) The projection coordinates of the left bionic eye camera on the normalized plane are calculated using the following formula.

[0027]

[0028]

[0029] In the above formula, ρ CN (·) represents the normalized planar projection equation of the bionic eye camera; m and n are the maximum pixel coordinates in the W and H directions of the pixel plane of the bionic eye camera, respectively;

[0030] S302, establish a panoramic camera coordinate system with the aperture center of the panoramic camera as the origin, and transform the four projected coordinates obtained in step S301 to the panoramic camera coordinate system using the following formula.

[0031]

[0032]

[0033] In the above formula, T L0P The pose of the panoramic camera relative to the initial pose coordinate system of the left bionic eye camera (established with the aperture center of the left bionic eye camera as the origin) is obtained through calibration; T L0LThe transformation matrix represents the pose of the left bionic eye camera relative to its initial pose coordinate system. This transformation matrix is ​​obtained through the bionic eye camera control feedback module.

[0034] S303, in the panoramic camera coordinate system, the origin of the initial pose coordinate system of the left bionic eye camera (the origin of the left bionic eye camera coordinate system is the aperture center of the left bionic eye camera) is connected to P. P(0,0) P P(m,0) P P(m,n) and P P(0,n) Connecting them separately yields four projected lines L representing the four vertices of the left bionic eye camera's field of view. P1 L P2 L P3 and L P4 The equations for the four projected lines are expressed as follows:

[0035]

[0036] In the above formula, P1 represents P(0,0); P2 represents P(m,0); P3 represents P(m,n); P4 represents P(0,n); PL0 represents the origin of the coordinate system of the initial pose coordinate system of the left bionic eye camera; a, b, and c are constant parameters, all representing the direction number of the straight line direction vector;

[0037] The normalized spherical equation for the panoramic camera is shown below.

[0038] x 2 +y 2 +z 2 =1

[0039] By simultaneously solving the equations of the projected line and the normalized sphere, the intersection points of the projected line and the normalized sphere of the panoramic camera are obtained. Since each line intersects the sphere at two points, but because the bionic eye camera is limited by its range of motion and cannot observe the scene within the panoramic camera coordinate system x < 0, only the four intersection points P where x > 0 are retained. P1 P P2 P P3 and P P4 The intersection point P P1 P P2 P P3 and P P4 That is, the projection of the vertex of the pixel plane of the left bionic eye camera onto the normalized sphere of the panoramic camera;

[0040] S304, using the normalized projection equation of the panoramic camera's sphere onto its pixel plane, the intersection point P is determined. P1 P P2 P P3 and P P4Projecting the image onto the pixel plane of the panoramic camera, we obtain the intersection point P. P1 P P2 P P3 and P P4 The projection point; the projection equation ρ of the normalized sphere of the panoramic camera onto its pixel plane. B→I (·) is shown below,

[0041] u PIi =ρ B→I (P Pi ), (i = 1, 2, 3, 4)

[0042] S305, connecting the intersection point P P1 P P2 P P3 and P P4 The projection point is used to obtain the left field of view projection area (that is, the projection of the left bionic eye camera pixel plane onto the pixel plane of the panoramic camera, which is also the projection onto the panoramic value image).

[0043] S306, the right bionic eye camera can obtain the right field of view projection area by following the above steps S301 to S305.

[0044] It should be noted that the left and right field-of-view projection areas are irregular quadrilaterals, which is a distortion problem that occurs during the spherical projection process of the panoramic camera.

[0045] More preferably, in step S300, the step of controlling the overlap of the left and right field-of-view projection areas specifically involves: using the bionic eye motion control module to control the movement of the left and right bionic eye cameras to overlap the left and right field-of-view projection areas as much as possible, i.e., controlling the two bionic eye cameras to approximately gaze at the same scene area. Furthermore, although the left and right field-of-view projection areas cannot completely overlap, they can be controlled to overlap as much as possible.

[0046] Preferably, in step S300, the specific steps for merging the left field of view projection area and the right field of view projection area are as follows: take the union of the left field of view projection area and the right field of view projection area to obtain the union area; then generate the maximum bounding rectangle of the union area to obtain the current binocular field of view projection area.

[0047] Preferably, in step S400, the specific steps for obtaining the average value of the current binocular field of view projection area in the panoramic value image are as follows: based on the pixel assignment results in the panoramic value image, calculate the average value of the pixel assignments in the current binocular field of view projection area to obtain the average value of the current binocular field of view projection area.

[0048] Preferably, the value threshold V th The selection is based on the number of feature points successfully tracked during the SLAM algorithm's operation, aiming to ensure the motion control module makes accurate motion decisions. If the average value of the current binocular field-of-view projection area is V... mc >V th This means that the current value of the binocular field-of-view projection area is sufficient to meet the requirements for the proper operation of the SLAM algorithm, even if the binocular field-of-view projection area is moved to the average value V. mc Further improvements in the region are limited, and motion-induced jitter may negatively impact SLAM performance. In such cases, it's advisable to avoid eye-shifting motion. If the average value V of the current binocular field of view projection area... mc ≤V th This means that the value in the scene within the binocular field of view projection area is insufficient to support the stable operation of the SLAM algorithm. This could be due to an insufficient number of extractable feature points in the scene or the presence of dynamic objects in the scene. In this case, the gaze is shifted to areas where the average value of the current scene is higher than the value threshold V. th And the target binocular field of view projection area that is closest to the current binocular field of view projection area.

[0049] Preferably, in step S500, the specific steps for searching for a target binocular field-of-view projection region in the panoramic value image that is higher than the value threshold are as follows:

[0050] S501, Generate a rectangular box in the panoramic value image that is the same size and position as the current binocular field of view projection area;

[0051] S502, translate the current rectangle from near to far with a step size s, and calculate the average value within the rectangle after each translation;

[0052] S503, compare the average value within the translated rectangle with the value threshold: if the average value within the translated rectangle is higher than the value threshold, then record the translated rectangle as the target binocular field of view projection area; if the average value within the translated rectangle is lower than or equal to the value threshold, then return to step S502 and update the current rectangle to the rectangle after the previous translation.

[0053] Preferably, in step S600, the displacement is an angular displacement, meaning the movement of the left and right bionic eye cameras is purely rotational without translation. Further, when the displacement is both angular and linear, the movement of the left and right bionic eye cameras includes both rotation and translation.

[0054] Preferably, in step S600, the step of moving the left and right bionic eye cameras according to the displacement amount specifically involves: controlling the yaw and pitch angles of the left and right bionic eye cameras respectively using motors to move the left and right bionic eye cameras. Further, the motors can be controlled by rotation commands issued by the bionic eye control module.

[0055] Preferably, in step S600, the step of calculating the displacement required by the left and right bionic eye cameras is as follows: using the normalized planar projection equation of the left or right bionic eye camera, the current pose of the left or right bionic eye camera in its initial pose coordinate system is calculated; using the normalized projection equation of the panoramic camera from the sphere to the pixel plane of the panoramic camera, the target pose of the left or right bionic eye camera as its field of view moves from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area is calculated; finally, based on the current pose and target pose of the left or right bionic eye camera, the rotation matrix R between the two poses is calculated. CT This allows us to obtain the displacement required for the left or right bionic eye camera to move from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area.

[0056] More preferably, in step S600, the specific steps for calculating the displacement required by the left and right bionic eye cameras are as follows:

[0057] S601 utilizes the projection equation ρ from the normalized sphere of the panoramic camera to its pixel plane. B→I (·) Determine the center pixel coordinates u of the target binocular field-of-view projection area in the panoramic value image. PT The target point is obtained by back-projecting it onto the normalized sphere of the panoramic camera; then, the target point is transformed into the initial pose coordinate system of the left bionic eye camera to obtain the target pose P of the left bionic eye camera. L0T The specific calculation formula is as follows:

[0058]

[0059] In the above formula, P L0T This indicates that the left bionic eye camera's field of view needs to be moved from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area; u PT The coordinates of the center pixel of the target binocular field-of-view projection area in the panoramic value map;

[0060] S602, utilizing the normalized plane projection equation ρ of the left bionic eye camera CN (·), the center point u of the left bionic eye camera pixel plane LC =[m / 2,n / 2] TBack-projecting onto the normalized plane of the left bionic eye camera yields the current pose P of the left bionic eye camera. L0C The specific calculation formula is as follows:

[0061]

[0062] In the above formula, P L0C This indicates the current pose of the left bionic eye camera in the initial pose coordinate system; u LC The center point of the pixel plane of the left bionic eye camera;

[0063] S603, using the current pose and target pose of the obtained left bionic eye camera, calculate the rotation matrix R between the two poses. CT Specifically:

[0064] First, according to the definition of the cross product, the rotation between two vectors is calculated using the axis of rotation.

[0065]

[0066] Since the rotation axis in the rotation vector is a unit vector, the vector after the cross product is normalized in the above formula;

[0067] Then calculate the rotation angle according to the following formula.

[0068]

[0069] Finally, based on the rotation axis n of the rotation vector CT and rotation angle θ CT The rotation matrix R is obtained using Rodriguez's formula. CT ,

[0070]

[0071] The angular displacement required to move the field of view of the left bionic eye camera from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area can be calculated.

[0072] S604, the right bionic eye camera can obtain the angular displacement required when the field of view of the right bionic eye camera moves from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area by following the above steps S601 to S603.

[0073] It should be noted that after the panoramic camera continues to capture the next image, the left and right bionic eye cameras will repeat the above positioning method to find the next target gaze area in order to obtain a high-value image that can be used as input for the SLAM system.

[0074] A second aspect of the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the active localization method for bionic binocular vision for performing SLAM tasks as described in the first aspect above.

[0075] A third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any step of the active localization method for performing a SLAM task using bionic binocular vision as described in the first aspect above.

[0076] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0077] (1) To address the problem of tracking failure caused by sparse textures or numerous dynamic objects when robots perform SLAM in unknown environments, this invention, based on the peripheral and central vision perception mechanisms of the human eye, uses a panoramic camera to simulate peripheral vision, proposes a panoramic value image based on the panoramic camera to guide the active gaze control of the binocular bionic eye, and proposes a binocular bionic eye gaze control strategy based on the panoramic value image to help obtain high-texture key information in the unknown full-scene visual information. Specifically: This invention first uses the FAST corner detection algorithm to extract feature points in the full scene, and simultaneously uses YOLOv7 to detect dynamic objects in the full scene; then, it uses pixel assignment to assign higher values ​​to feature points and surrounding pixels in the panoramic image, with the value gradually decreasing as the neighborhood radius increases, and assigning low values ​​to all pixels in the target box of dynamic object detection, thereby obtaining a panoramic value image; then, it uses the normalized plane projection equation of the bionic eye camera and the projection equation of the spherical projection of the panoramic camera onto the pixel plane to calculate the projection of the area gazed by the bionic eye camera in the panoramic value image; and then uses the value threshold V defined by the system. th The value of the scene in the currently viewed area is evaluated; finally, the rotational motion required for the bionic eye to move from the current pose to the target pose is calculated using the Rodriguez formula. In one embodiment, the localization method of the present invention enables the robot trajectory to be continuous without tracking failure, effectively improving the stability of the SLAM system.

[0078] (2) This invention utilizes panoramic images captured by a panoramic camera to simulate the peripheral vision of the human eye. High-value images are obtained through custom assignment of values ​​to the panoramic images, and then the binocular camera continuously focuses on the high-value areas. This improves the system's ability to acquire high-value information in a scene. Especially when performing SLAM tasks, it avoids the tracking loss problem that often occurs when the images captured by the camera have sparse textures or many dynamic objects, thus improving the stability of the SLAM system and enabling the visual SLAM method to achieve accurate localization and mapping in unknown environments. This invention has significant practical application value and can be widely applied in fields such as robot navigation and autonomous driving, providing strong support for technological innovation and progress in related industries.

[0079] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0080] Figure 1 This is a flowchart illustrating the active localization method for binocular vision in the bionic eye of the present invention.

[0081] Figure 2 This is a panoramic value image in Embodiment 1 of the present invention; in the figure, the box represents the detected dynamic object, and all pixels within it are assigned low pixel values.

[0082] Figure 3 This is a schematic diagram of the panoramic value image-guided line-of-sight control of the left and right bionic eyes according to the present invention; in the figure, the green rectangle is the current binocular field of view projection area of ​​the left and right bionic eye cameras, and the red rectangle is the target binocular field of view projection area of ​​the left and right bionic eye cameras.

[0083] Figure 4 This is an experimental result diagram of the robot performing a SLAM task in a bush scene in Embodiment 1 of the present invention. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0085] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0086] Example 1

[0087] This embodiment provides an active localization method using bionic binocular vision for performing SLAM tasks, comprising the following steps:

[0088] The S100 uses a panoramic camera capable of acquiring 360° scene information to obtain panoramic images; then it detects key scene information in the panoramic images that affects the accuracy and robustness of SLAM operation, such as feature points and dynamic objects.

[0089] The feature point extraction method employs the FAST keypoint detection algorithm. During feature point extraction, since feature matching is not required, the calculation of the BRIEF descriptor in the ORB feature points extracted by the SLAM algorithm is omitted. Furthermore, FAST is a corner detection algorithm, primarily used to detect areas with significant local pixel grayscale changes, and is known for its high speed. Its core idea is: if a pixel differs significantly from its neighboring pixels (too bright or too dark), it is more likely to be a corner point. The detection process of the FAST keypoint detection algorithm is as follows:

[0090] (1) Select pixel p in the image, and assume its brightness is I. p ;

[0091] (2) Set a threshold T(I) p 20% of;

[0092] (3) Select 16 pixels on a circle with a radius of 3, with pixel p as the center;

[0093] (4) If the selected circle has N consecutive points with a brightness greater than I p +T or less than I p - The pixel p in T can be considered a feature point (N is usually 12, i.e., FAST-12);

[0094] (5) Repeat the above 4 steps and perform the same operation on each pixel.

[0095] The method for detecting dynamic objects is to use the YOLO v7 detection algorithm; the dynamic objects are represented by detection bounding boxes. For dynamic object detection, the YOLO v7 detection algorithm is used to detect dynamic objects such as pedestrians and animals in the panoramic image, or potential dynamic objects that have a negative impact on map reuse.

[0096] S200 assigns values ​​to pixels in the panoramic image based on key scene information to obtain a panoramic value image, such as... Figure 2 As shown.

[0097] The specific steps for assigning values ​​to pixels in the panoramic image based on key scene information are as follows: Pixels located at feature points and pixels in the neighborhood of feature points in the panoramic image are assigned high grayscale values, while all pixels within the detection target box of dynamic objects are assigned low grayscale values. The grayscale value assigned to pixels in the neighborhood of feature points decreases as the neighborhood radius increases. If the same pixel is assigned a value multiple times, the pixel takes the maximum value assigned to it.

[0098] Furthermore, since feature points are the primary focus of the SLAM algorithm, feature points and their neighborhoods are assigned high grayscale values; the standard for high grayscale values ​​is a grayscale value higher than 240. For dynamic objects that severely degrade the performance of the SLAM algorithm, all pixels in the detection bounding box of the dynamic object are assigned low grayscale values, even lower than areas where no feature points were detected; the standard for low grayscale values ​​is a grayscale value lower than 80.

[0099] S300 projects the field of view regions of the left and right bionic eye cameras onto the panoramic value image, respectively, to obtain the left and right field of view projection regions; simultaneously, it controls the overlap of the left and right field of view projection regions; and finally merges the left and right field of view projection regions to obtain the current binocular field of view projection region, such as... Figure 3 As shown in the green rectangle.

[0100] Specifically, the step of projecting the field of view regions of the left and right bionic eye cameras onto the panoramic value image is as follows: using the normalized planar projection equation of the bionic eye camera and the normalized spherical projection equation of the panoramic camera, the projection of the field of view regions (pixel planes) of the left and right bionic eye cameras onto the panoramic value image (pixel plane) of the panoramic camera is calculated respectively.

[0101] Furthermore, the specific steps for calculating the projection of the pixel planes of the left and right bionic eye cameras onto the pixel plane of the panoramic camera are as follows:

[0102] S301, using the normalized plane projection equation of the bionic eye camera, calculates the four pixels u located at the vertices of the pixel plane of the left bionic eye camera. L(0,0) u L(m,0) u L(m,n) and u L(0,n) The projection coordinates of the left bionic eye camera on the normalized plane are calculated using the following formula.

[0103]

[0104]

[0105] In the above formula, ρ CN(·) represents the normalized planar projection equation of the bionic eye camera; m and n are the maximum pixel coordinates in the W and H directions of the pixel plane of the bionic eye camera, respectively;

[0106] S302, establish a panoramic camera coordinate system with the aperture center of the panoramic camera as the origin, and transform the four projected coordinates obtained in step S301 to the panoramic camera coordinate system using the following formula.

[0107]

[0108]

[0109] In the above formula, T L0P The pose of the panoramic camera relative to the initial pose coordinate system of the left bionic eye camera (established with the aperture center of the left bionic eye camera as the origin) is obtained through calibration; T L0L The transformation matrix represents the pose of the left bionic eye camera relative to its initial pose coordinate system. This transformation matrix is ​​obtained through the bionic eye camera control feedback module.

[0110] S303, in the panoramic camera coordinate system, the origin of the initial pose coordinate system of the left bionic eye camera (the origin of the left bionic eye camera coordinate system is the aperture center of the left bionic eye camera) is connected to P. P(0,0) P P(m,0) P P(m,n) and P P(0,n) Connecting them separately yields four projected lines L representing the four vertices of the left bionic eye camera's field of view. P1 L P2 L P3 and L P4 The equations for the four projected lines are expressed as follows:

[0111]

[0112] In the above formula, P1 represents P(0,0); P2 represents P(m,0); P3 represents P(m,n); P4 represents P(0,n); PL0 represents the origin of the coordinate system of the initial pose coordinate system of the left bionic eye camera; a, b, and c are constant parameters, all representing the direction number of the straight line direction vector;

[0113] The normalized spherical equation for the panoramic camera is shown below.

[0114] x 2 +y 2 +z 2 =1

[0115] By simultaneously solving the equations of the projected line and the normalized sphere, the intersection points of the projected line and the normalized sphere of the panoramic camera are obtained. Since each line intersects the sphere at two points, but because the bionic eye camera is limited by its range of motion and cannot observe the scene within the panoramic camera coordinate system x < 0, only the four intersection points P where x > 0 are retained. P1 P P2 P P3 and P P4 The intersection point P P1 P P2 P P3 and P P4 That is, the projection of the vertex of the pixel plane of the left bionic eye camera onto the normalized sphere of the panoramic camera;

[0116] S304, using the normalized projection equation of the panoramic camera's sphere onto its pixel plane, the intersection point P is determined. P1 P P2 P P3 and P P4 Projecting the image onto the pixel plane of the panoramic camera, we obtain the intersection point P. P1 P P2 P P3 and P P4 The projection point; the projection equation ρ of the normalized sphere of the panoramic camera onto its pixel plane. B→I (·) is shown below,

[0117] u PIi =ρ B→I (P Pi ), (i = 1, 2, 3, 4)

[0118] S305, connecting the intersection point P P1 P P2 P P3 and P P4 The projection point is used to obtain the left field of view projection area (that is, the projection of the left bionic eye camera pixel plane onto the pixel plane of the panoramic camera, which is also the projection onto the panoramic value image).

[0119] S306, the right bionic eye camera can obtain the right field of view projection area by following the above steps S301 to S305.

[0120] It should be noted that the left and right field-of-view projection areas are irregular quadrilaterals, which is a distortion problem that occurs during the spherical projection process of the panoramic camera.

[0121] The specific steps for controlling the overlap of the left and right field-of-view projection areas are as follows: the bionic eye motion control module controls the movement of the left and right bionic eye cameras to overlap the left and right field-of-view projection areas as much as possible, that is, to control the two bionic eye cameras to approximately gaze at the same scene area. Furthermore, although the left and right field-of-view projection areas cannot completely overlap, they can be controlled to overlap as much as possible.

[0122] Furthermore, the specific steps for merging the left and right field-of-view projection regions are as follows: take the union of the left and right field-of-view projection regions to obtain the union region; then generate the largest bounding rectangle of the union region to obtain the current binocular field-of-view projection region.

[0123] S400, obtain the average value of the current binocular field of view projection area in the panoramic value image, specifically: calculate the average value of the pixel points in the current binocular field of view projection area based on the pixel point assignment results in the panoramic value image, and obtain the average value of the current binocular field of view projection area.

[0124] Wherein, the value threshold V th The selection is based on the number of feature points successfully tracked during the SLAM algorithm's operation, aiming to ensure the motion control module makes accurate motion decisions. If the average value of the current binocular field-of-view projection area is V... mc >V th This means that the current value of the binocular field-of-view projection area is sufficient to meet the requirements for the proper operation of the SLAM algorithm, even if the binocular field-of-view projection area is moved to the average value V. mc Further improvements in the region are limited, and motion-induced jitter may negatively impact SLAM performance. In such cases, it's advisable to avoid eye-shifting motion. If the average value V of the current binocular field of view projection area... mc ≤V th This means that the value in the scene within the binocular field of view projection area is insufficient to support the stable operation of the SLAM algorithm. This could be due to an insufficient number of extractable feature points in the scene or the presence of dynamic objects in the scene. In this case, the gaze is shifted to areas where the average value of the current scene is higher than the value threshold V. th And the target binocular field of view projection area that is closest to the current binocular field of view projection area.

[0125] S500, determine the value mean of the current binocular field-of-view projection area and the value threshold: If the value mean of the current binocular field-of-view projection area is greater than the value threshold, do not move the current binocular field-of-view projection area, and use the high-value images currently acquired by the left and right bionic eye cameras as input to the SLAM system; if the value mean of the current binocular field-of-view projection area is less than or equal to the value threshold, search for a target binocular field-of-view projection area in the panoramic value image whose value mean is higher than the value threshold (e.g., ...). Figure 3 (As shown in the red rectangle in the middle), then proceed to step S600.

[0126] The specific steps for searching for a target binocular field-of-view projection region in the panoramic value image that is higher than the value threshold are as follows:

[0127] S501, Generate a rectangular box in the panoramic value image that is the same size and position as the current binocular field of view projection area;

[0128] S502, translate the current rectangle from near to far with a step size s, and calculate the average value within the rectangle after each translation;

[0129] S503, compare the average value within the translated rectangle with the value threshold: if the average value within the translated rectangle is higher than the value threshold, then record the translated rectangle as the target binocular field of view projection area; if the average value within the translated rectangle is lower than or equal to the value threshold, then return to step S502 and update the current rectangle to the rectangle after the previous translation.

[0130] S600 calculates the required displacement of the left and right bionic eye cameras based on the current binocular field of view projection area and the target binocular field of view projection area; moves the left and right bionic eye cameras according to the displacement, and then uses the high-value images currently acquired by the left and right bionic eye cameras as the input of the SLAM system.

[0131] The displacement is angular displacement, meaning that the movement of the two bionic eye cameras is pure rotation without translation.

[0132] The steps for calculating the angular displacement required for the left and right bionic eye cameras are as follows: Using the normalized planar projection equation of the left or right bionic eye camera, calculate the current pose of the left or right bionic eye camera in its initial pose coordinate system; using the normalized projection equation from the normalized sphere to the pixel plane of the panoramic camera, calculate the target pose of the left or right bionic eye camera as its field of view moves from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area; finally, based on the current pose and target pose of the left or right bionic eye camera, calculate the rotation matrix R between the two poses. CTThis allows us to obtain the angular displacement required for the left or right bionic eye camera to move from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area.

[0133] Furthermore, the specific steps for calculating the required displacement of the left and right bionic eye cameras are as follows:

[0134] S601 utilizes the projection equation ρ from the normalized sphere of the panoramic camera to its pixel plane. B→I (·) Determine the center pixel coordinates u of the target binocular field-of-view projection area in the panoramic value image. PT The target point is obtained by back-projecting it onto the normalized sphere of the panoramic camera; then, the target point is transformed into the initial pose coordinate system of the left bionic eye camera to obtain the target pose P of the left bionic eye camera. L0T The specific calculation formula is as follows:

[0135]

[0136] In the above formula, P L0T This indicates that the left bionic eye camera's field of view needs to be moved from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area; u PT The coordinates of the center pixel of the target binocular field-of-view projection area in the panoramic value map;

[0137] S602, utilizing the normalized plane projection equation ρ of the left bionic eye camera CN (·), the center point u of the left bionic eye camera pixel plane LC =[m / 2,n / 2] T Back-projecting onto the normalized plane of the left bionic eye camera yields the current pose P of the left bionic eye camera. L0C The specific calculation formula is as follows:

[0138]

[0139] In the above formula, P L0C This indicates the current pose of the left bionic eye camera in the initial pose coordinate system; u LC The center point of the pixel plane of the left bionic eye camera;

[0140] S603, using the current pose and target pose of the obtained left bionic eye camera, calculate the rotation matrix R between the two poses. CT Specifically:

[0141] First, according to the definition of the cross product, the rotation between two vectors is calculated using the axis of rotation.

[0142]

[0143] Since the rotation axis in the rotation vector is a unit vector, the vector after the cross product is normalized in the above formula;

[0144] Then calculate the rotation angle according to the following formula.

[0145]

[0146] Finally, based on the rotation axis n of the rotation vector CT and rotation angle θ CT The rotation matrix R is obtained using Rodriguez's formula. CT ,

[0147]

[0148] The angular displacement required to move the field of view of the left bionic eye camera from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area can be calculated.

[0149] S604, the right bionic eye camera can obtain the angular displacement required when the field of view of the right bionic eye camera moves from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area by following the above steps S601 to S603.

[0150] The specific steps for moving the left and right bionic eye cameras according to the displacement are as follows: The left and right bionic eye cameras are moved by controlling the yaw and pitch angles of the two cameras respectively using motors. Furthermore, the motors can be controlled by rotation commands issued by the bionic eye control module.

[0151] It should be noted that after the panoramic camera continues to capture the next image, the left and right bionic eye cameras will repeat the above positioning method to reposition themselves in order to obtain high-value images that can be used as input for the SLAM system.

[0152] Figure 4 This is an octree map built by the robot when performing a SLAM task in a bush after using the localization method of this invention. Figure 4 As can be seen, the mobile robot completed a full loop in the complex environment of the bushes, with a continuous trajectory and no trajectory breakpoints, indicating that the SLAM tracking was not lost. The same result was maintained when the experiment was repeated in four other different bushes. In contrast, the robot that did not use the localization method of this invention experienced trajectory breakpoints twice when performing the SLAM task in the above five bushes. Therefore, the localization method of this invention has achieved the technical effect of improving the stability of SLAM tracking.

[0153] Example 2

[0154] An electronic device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements an active localization method for performing SLAM tasks using bionic binocular vision, as described in Embodiment 1.

[0155] Example 3

[0156] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the active localization method for performing SLAM tasks using bionic binocular vision as described in Embodiment 1.

[0157] In summary, this invention effectively overcomes the shortcomings of the prior art and has high industrial applicability. The above embodiments are intended to illustrate the substantive content of this invention, but are not intended to limit the scope of protection of this invention. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this invention without departing from the essence and scope of protection of this invention.

Claims

1. An active localization method using bionic binocular vision for performing SLAM tasks, characterized in that, Includes the following steps: The S100 uses a panoramic camera to acquire panoramic images, and then detects key scene information in the panoramic images that affects the execution of the SLAM task. S200: Based on key scene information, values ​​are assigned to pixels in the panoramic image to obtain a panoramic value image. S300 projects the field of view of the left and right bionic eye cameras onto the panoramic value image, respectively, to obtain the left field of view projection area and the right field of view projection area; at the same time, it controls the overlap of the left field of view projection area and the right field of view projection area. Merge the left and right field-of-view projection areas to obtain the current binocular field-of-view projection area; S400, obtains the average value of the current binocular field-of-view projection area in the panoramic value image; S500, determine the value mean of the current binocular field of view projection area and the value threshold: if the value mean of the current binocular field of view projection area is greater than the value threshold, do not move the current binocular field of view projection area, and use the high-value images currently acquired by the left and right bionic eye cameras as the input of the SLAM system; if the value mean of the current binocular field of view projection area is less than or equal to the value threshold, search for a target binocular field of view projection area with a higher value than the value threshold in the panoramic value image, and then proceed to step S600; S600: Based on the current binocular field of view projection area and the target binocular field of view projection area, calculate the required displacement of the left and right bionic eye cameras respectively; move the left and right bionic eye cameras respectively according to the displacement amount, and finally use the high-value images acquired by the left and right bionic eye cameras in the target binocular field of view projection area as the input of the SLAM system.

2. The active positioning method according to claim 1, characterized in that, The key scene information includes feature points and dynamic objects; the feature points are extracted using the FAST key point detection algorithm. The detection process for the dynamic object involves using the YOLO v7 detection algorithm to detect the dynamic object and obtain its detection bounding box.

3. The active positioning method according to claim 2, characterized in that, In step S200, the step of assigning values ​​to pixels in the panoramic image based on key scene information specifically involves: assigning high pixel values ​​to pixels located at feature points and pixels located in the neighborhood of feature points in the panoramic image, and assigning low pixel values ​​to all pixels located within the detection target box of dynamic objects; wherein, the pixel value assigned to pixels located in the neighborhood of feature points decreases as the neighborhood radius increases; if the same pixel is assigned multiple values, the pixel takes the maximum value assigned to it.

4. The active positioning method according to claim 3, characterized in that, In step S300, the step of projecting the field of view regions of the left and right bionic eye cameras onto the panoramic value image is specifically as follows: using the normalized planar projection equation of the bionic eye camera and the normalized spherical projection equation of the panoramic camera, the projection of the field of view regions of the left and right bionic eye cameras onto the panoramic value image of the panoramic camera is calculated respectively.

5. The active positioning method according to claim 4, characterized in that, In step S300, the specific steps for merging the left field of view projection area and the right field of view projection area are as follows: take the union of the left field of view projection area and the right field of view projection area to obtain the union area; then generate the maximum bounding rectangle of the union area to obtain the current binocular field of view projection area.

6. The active positioning method according to claim 5, characterized in that, In step S400, the specific steps for obtaining the average value of the current binocular field of view projection area in the panoramic value image are as follows: based on the pixel assignment results in the panoramic value image, calculate the average value of the pixel assignments in the current binocular field of view projection area to obtain the average value of the current binocular field of view projection area.

7. The active positioning method according to claim 6, characterized in that, In step S500, the specific steps for searching for a target binocular field-of-view projection region in the panoramic value image that is higher than the value threshold are as follows: S501, Generate a rectangular box in the panoramic value image that is the same size and position as the current binocular field of view projection area; S502, translate the current rectangle from near to far with a step size s, and calculate the average value within the rectangle after each translation; S503, compare the average value within the translated rectangle with the value threshold: if the average value within the translated rectangle is higher than the value threshold, then record the translated rectangle as the target binocular field of view projection area; if the average value within the translated rectangle is lower than or equal to the value threshold, then return to step S502 and update the current rectangle to the rectangle after the previous translation.

8. The active positioning method according to claim 7, characterized in that, In step S600, the displacement is angular displacement; the step of calculating the displacement required for the left and right bionic eye cameras is as follows: using the normalized plane projection equation of the left or right bionic eye camera, calculate the current pose of the left or right bionic eye camera in its initial pose coordinate system. By using the projection equation from the normalized sphere of the panoramic camera to the pixel plane of the panoramic camera, the target pose of the left or right bionic eye camera as its field of view moves from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area is calculated. Finally, based on the current pose and target pose of the left or right bionic eye camera, the rotation matrix between the two poses is calculated. R CT This allows us to obtain the displacement required for the left or right bionic eye camera to move from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the active localization method for bionic binocular vision used in performing SLAM tasks as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the active localization method for bionic binocular vision used in performing SLAM tasks as described in any one of claims 1-8.

Images