A method, device, equipment and storage medium for determining the target pose based on binocular vision

Abstract

The present disclosure relates to a method, apparatus, device, and storage medium for determining the target pose based on trinocular vision. The method includes: obtaining a first image, a second image, and a third image, where the first image, the second image, and the third image are images obtained by a first camera, a second camera, and a third camera respectively for the same target. The target area in the first image is aligned with the target area in the second image in a first direction, and the target area in the first image is aligned with the target area in the third image in a second direction, and the first direction is perpendicular to the second direction; determining the centerlines of the target areas in the first image, the second image, and the third image; determining at least four groups of corresponding point pairs based on the centerlines and contours of the target areas; and determining the target pose based on the at least four groups of corresponding point pairs. In this way, the target pose can be determined without constructing a dense point cloud of the object, reducing the amount of data to be processed and improving the speed of pose estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and particularly to a method, device, equipment and storage medium for determining the target pose based on binocular vision. Background Art

[0002] Binocular vision technology refers to simultaneously capturing the same scene with three cameras, and obtaining information such as the depth information of the scene, the position and size of the target object through image processing algorithms. The pose estimation algorithm plays an important role in binocular vision technology. It can estimate the position and orientation of the target object based on the captured scene image, so as to help users better understand the position and orientation of the target object in the scene.

[0003] The prior art usually needs to first construct a dense point cloud of the object. However, constructing a dense point cloud requires processing a large amount of data, resulting in slow point cloud imaging and high requirements for the light environment where the object is located. In addition, in the stereo matching stage, a large amount of calculations are required to find corresponding point pairs (for example, matching cost calculation, cost aggregation, etc.), and the algorithm complexity is high, so it cannot be stably used outdoors. Summary of the Invention

[0004] According to a first aspect of the present disclosure, there is provided a method for determining the target pose based on binocular vision, including: obtaining a first image, a second image and a third image; wherein, the first image, the second image and the third image are respectively images captured by a first camera, a second camera and a third camera for the same target, the target area in the first image is aligned with the target area in the second image in a first direction, the target area in the first image is aligned with the target area in the third image in a second direction, and the first direction is perpendicular to the second direction; determining the center lines of the respective target areas in the first image, the second image and the third image; determining at least four groups of corresponding point pairs based on the center lines and the contours of the respective target areas; and determining the target pose based on the at least four groups of corresponding point pairs.

[0005] According to a second aspect of the present disclosure, there is provided an apparatus for determining an object pose based on binocular vision, including: an acquisition module configured to acquire a first image, a second image, and a third image; wherein the first image, the second image, and the third image are respectively images of the same object captured by a first camera, a second camera, and a third camera, the target region in the first image is aligned with the target region in the second image in a first direction, the target region in the first image is aligned with the target region in the third image in a second direction, and the first direction is perpendicular to the second direction; a centerline determination module configured to determine the centerlines of the respective target regions in the first image, the second image, and the third image; a corresponding point pair determination module configured to determine at least four groups of corresponding point pairs based on the centerlines and the contours of the respective target regions; and a pose determination module configured to determine the object pose based on the at least four groups of corresponding point pairs.

[0006] According to a third aspect of the present disclosure, there is provided an electronic device, including: at least one processor; a memory configured to store executable instructions for the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the method as described in the first aspect of the present disclosure.

[0007] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method as described in the first aspect of the present disclosure is implemented.

[0008] The method for determining an object pose based on binocular vision provided in the embodiments of the present disclosure includes: acquiring a first image, a second image, and a third image; wherein the first image, the second image, and the third image are respectively images of the same object captured by a first camera, a second camera, and a third camera, the target region in the first image is aligned with the target region in the second image in a first direction, the target region in the first image is aligned with the target region in the third image in a second direction, and the first direction is perpendicular to the second direction; determining the centerlines of the respective target regions in the first image, the second image, and the third image; determining at least four groups of corresponding point pairs based on the centerlines and the contours of the respective target regions; and determining the object pose based on the at least four groups of corresponding point pairs. Compared with the prior art, the present disclosure can determine at least four groups of corresponding point pairs based on the centerlines and the contours of the respective target regions in the first image, the second image, and the third image, eliminating the stereo matching process, reducing the computational amount, and having a low algorithm complexity; secondly, the present disclosure determines the object pose based on the at least four groups of corresponding point pairs determined above, without the need to construct a dense point cloud of the object, reducing the amount of data to be processed, and improving the speed of pose estimation, so that it can be stably used outdoors. Description of the Drawings

[0009] In the following description of exemplary embodiments with reference to the accompanying drawings, more details, features, and advantages of the present disclosure are disclosed. In the drawings:

[0010] Figure 1 Flowchart of a method for determining the target pose based on binocular vision provided for an exemplary embodiment of the present disclosure;

[0011] Figure 2 Flowchart of a method for determining the target exposure parameters of the first camera, the second camera, or the third camera provided for an exemplary embodiment of the present disclosure;

[0012] Figure 3 Flowchart of a method for determining at least four sets of homologous point pairs provided for an exemplary embodiment of the present disclosure;

[0013] Figure 4 Flowchart of a method for determining the centerlines of the target regions in the first image, the second image, and the third image provided for another exemplary embodiment of the present disclosure;

[0014] Figure 5 Flowchart of a method for determining at least four sets of homologous point pairs provided for another exemplary embodiment of the present disclosure;

[0015] Figure 6 Flowchart of a method for determining the target pose provided for an exemplary embodiment of the present disclosure;

[0016] Figure 7 Schematic block diagram of a device for determining the target pose based on binocular vision provided for an exemplary embodiment of the present disclosure;

[0017] Figure 8 Schematic block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0018] Figure 9 Structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. Detailed Description of the Invention

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0020] It should be understood that the various steps described in the method embodiments of the present disclosure may be executed in different orders and / or executed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0021] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependent relationships.

[0022] It should be noted that the modifications of "one" and "plural" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure.

[0025] Figure 1 A flowchart of a method for determining the target pose based on binocular vision provided for an exemplary embodiment of the present disclosure is as Figure 1 shown, and includes the following steps:

[0026] Step S102: Obtain a first image, a second image, and a third image; wherein, the first image, the second image, and the third image are images obtained by a first camera, a second camera, and a third camera respectively for the same target, the target area in the first image is aligned with the target area in the second image in a first direction, the target area in the first image is aligned with the target area in the third image in a second direction, and the first direction is perpendicular to the second direction.

[0027] The embodiments of the present disclosure do not limit the positional relationship among the above-mentioned first camera, second camera, and third camera. In a feasible implementation manner, the above-mentioned first camera, second camera, and third camera may be arranged in a spatial triangle. Optionally, the above-mentioned first camera, second camera, and third camera may be arranged in a right triangle, and the above-mentioned first camera is located at the vertex of the right angle in the right triangle. The above-mentioned configuration manner of the right triangle can make the calibration accuracy higher, that is, the positional relationship among the three cameras obtained through calibration is more accurate. In this implementation manner, the above-mentioned first camera is horizontally aligned with the above-mentioned second camera, and the above-mentioned first camera is vertically aligned with the above-mentioned third camera; the above-mentioned first camera and the above-mentioned second camera form a binocular in the horizontal direction, and the above-mentioned first camera and the above-mentioned third camera form a binocular in the vertical direction. It can be understood that the above-mentioned horizontal alignment of the first camera and the second camera means that the optical centers of the first camera and the second camera are aligned in the horizontal direction; the above-mentioned vertical alignment of the first camera and the third camera means that the optical centers of the first camera and the third camera are aligned in the vertical direction.

[0028] In a feasible implementation manner, before obtaining the first image, second image, and third image, the method further includes: calibrating the first camera, second camera, and third camera to obtain the calibration parameters of the first camera, second camera, and third camera; where the calibration parameters include the internal parameters, external parameters, distortion parameters of the camera, and the positional relationship among the first camera, second camera, and third camera. The internal parameters of the camera are the parameters describing the camera imaging model, including the focal length, principal point coordinates, etc., and the external parameters of the camera are the parameters describing the conversion relationship from the world coordinate system to the camera coordinate system, including the rotation matrix and translation vector, etc. In this implementation manner, the internal parameters of the above-mentioned first camera, second camera, and third camera may include the focal lengths and principal point coordinates of the first camera, second camera, and third camera. The positional relationship among the first camera, second camera, and third camera may include the transformation matrix from the first camera to the third camera, the transformation matrix from the second camera to the first camera, and the transformation matrix from the second camera to the third camera.

[0029] In a feasible implementation manner, before obtaining the first image, second image, and third image, the method further includes: determining the target exposure parameters of the first camera, second camera, or third camera. This implementation manner does not limit the determination of the target exposure parameters of the first camera, second camera, or third camera. Optionally, as Figure 2 shown, the determination of the target exposure parameters of the first camera, second camera, or third camera includes:

[0030] Step S202: Obtain multiple reference images. Among them, the multiple reference images are images obtained by the first camera, the second camera, or the third camera shooting the same target under multiple sets of preset exposure parameters. The multiple reference images include a first reference image, and the first reference image is an image obtained when the fill light of the camera is not turned on.

[0031] Step S204: Fuse the multiple reference images to obtain a second reference image.

[0032] Step S206: Determine a first gray value and a second gray value. The first gray value is the average gray value of the target area of the first reference image, and the second gray value is the average gray value of the target area of the second reference image. The target area of the first reference image and the target area of the second reference image correspond to the same area of the target.

[0033] Step S208: Based on the first gray value and the second gray value, determine the target exposure parameter of the first camera, the second camera, or the third camera.

[0034] In this embodiment, first, since the first reference image is obtained when the fill light is not turned on, and the first gray value is the average gray value of the target area of the first reference image, the first gray value can directly reflect the light intensity of the ambient light in the scene where the target is located. Second, since the second reference image is formed by fusing the multiple images, and the second gray value is the average gray value of the target area of the second reference image, the second gray value can reflect the sensitivity of the target to the fill light in this scene. Finally, according to the first gray value and the second gray value, the target exposure parameter of the camera is determined. This target exposure parameter is more suitable for the ambient light and target requirements of the current scene, and has the advantages of high stability, accurate measurement, and strong ambient light adaptation ability.

[0035] The embodiments of the present disclosure do not limit the specific directions of the above-mentioned first direction and second direction, as long as the first direction is perpendicular to the second direction. In a feasible embodiment, the first direction may be the horizontal direction of the image, and the second direction may be the vertical direction of the image. In this embodiment, the target area in the first image is aligned with the target area in the second image in the first direction, and the target area in the first image is aligned with the target area in the third image in the second direction, which means that the target area in the first image is aligned with the target area in the second image in the horizontal direction of the image, and the target area in the first image is aligned with the target area in the third image in the vertical direction of the image. It can be understood that the target area in the first image is aligned with the target area in the second image in the horizontal direction of the image, and the target area in the first image is aligned with the target area in the third image in the vertical direction of the image, indicating that the epipolar lines of the first image and the second image are parallel to the coordinate axis in the horizontal direction of the image, and the epipolar lines of the first image and the third image are parallel to the coordinate axis in the vertical direction of the image, and the horizontal parallax between the first image and the second image is equal to the vertical parallax between the first image and the third image. It can also be considered that the row coordinates of the pixel points in the first image are the same as those of the corresponding pixel points in the second image, and the column coordinates of the pixel points in the first image are the same as those of the corresponding pixel points in the third image.

[0036] The embodiments of the present disclosure do not limit the specific manner of obtaining the first image, the second image, and the third image. In a feasible embodiment, obtaining the first image, the second image, and the third image includes:

[0037] Obtaining a first original image, a second original image, and a third original image; wherein, the first original image, the second original image, and the third original image are original images obtained by a first camera, a second camera, and a third camera respectively for the same target;

[0038] Performing stereo rectification on the first original image, the second original image, and the third original image to obtain the first image, the second image, and the third image.

[0039] The embodiments of the present disclosure do not limit the specific manner of performing stereo rectification on the first original image, the second original image, and the third original image, as long as the target area in the first image is aligned with the target area in the second image in the first direction, the target area in the first image is aligned with the target area in the third image in the second direction, and the first direction is perpendicular to the second direction.

[0040] In an alternative embodiment, the stereoscopic correction of the first original image, the second original image, and the third original image to obtain the first image, the second image, and the third image may include:

[0041] 1) Based on the calibration parameters, use the Bouguet algorithm to horizontally correct the first original image I1 and the second original image I2 to obtain row-aligned images I1' and I2'; where I1' = I1.

[0042] Since the first original image is used as a reference image, it can be considered that I1' = I1. The Bouguet correction algorithm is an image correction algorithm designed to perform geometric correction and optical distortion correction on a photographic device to improve the quality and accuracy of the image. This algorithm was proposed by Jean-Yves Bouguet in 1999 and is currently widely used in the fields of computer vision and robotics.

[0043] 2) Based on the calibration parameters, use the Bouguet algorithm to correct the first original image I1 and the third original image I3 to obtain the overall rotation matrices of the first camera and the third camera respectively;

[0044] 3) Based on the overall rotation matrices of the first camera and the third camera, rotate the third original image around the optical center of the third camera to obtain column-aligned images I1" and I3'; where I1" = I1. Since the first original image is used as a reference image, it can be considered that I1" = I1;

[0045] 4) Determine images I1, I2', and I3' as the first corrected image, the second corrected image, and the third corrected image respectively;

[0046] 5) Obtain the feature points of the first corrected image, the second corrected image, and the third corrected image respectively,

[0047] 6) Optimize the first corrected image, the second corrected image, and the third corrected image based on the feature points to obtain the first image, the second image, and the third image.

[0048] In another alternative embodiment, the stereoscopic correction of the first original image, the second original image, and the third original image to obtain the first image, the second image, and the third image may further include:

[0049] Correct the first original image, the second original image, and the third original image based on the distortion parameters of the first camera, the second camera, and the third camera to obtain the first distortion-corrected image, the second distortion-corrected image, and the third distortion-corrected image;

[0050] Perform epipolar correction in the horizontal direction on the above-mentioned first distorted correction image and second distorted correction image to obtain the above-mentioned first correction image and second correction image;

[0051] Using the first correction image as a reference image, perform epipolar correction in the vertical direction on the above-mentioned third distorted image to obtain a third correction image;

[0052] Determine the above-mentioned first correction image, second correction image, and third correction image as the first image, second image, and third image.

[0053] Step S104: Determine the centerlines of the target regions in the above-mentioned first image, second image, and third image.

[0054] Step S106: Based on the centerlines of the above-mentioned target regions and the contours of the above-mentioned target regions, determine at least four groups of corresponding point pairs.

[0055] In a feasible implementation manner, before determining the centerlines of the target regions in the above-mentioned first image, second image, and third image, it further includes: respectively determining the target regions in the above-mentioned first image, second image, and third image; determining the centerlines of the above-mentioned target regions. This implementation manner does not limit the specific manner of respectively determining the target regions in the above-mentioned first image, second image, and third image. Optionally, a trained neural network can be used to respectively determine the target regions in the above-mentioned first image, second image, and third image. Optionally, the above-mentioned neural network can be YOLO, SSD, RetinaNet, R-CNN, Fast R-CNN, or Faster R-CNN, etc.

[0056] The embodiments of the present disclosure do not limit the specific manner of determining the centerlines of the target regions in the above-mentioned first image, second image, and third image. In an alternative implementation manner, determining the centerlines of the target regions in the above-mentioned first image, second image, and third image includes:

[0057] When the contours of the above-mentioned target regions are rectangles, determine a first centerline and a second centerline; wherein, the above-mentioned first centerline is the centerline of the target regions in the first image and the second image in the first direction, and the above-mentioned second centerline is the centerline of the target regions in the first image and the third image in the second direction.

[0058] The embodiments of the present disclosure do not limit the specific manner of determining at least four groups of corresponding point pairs based on the centerlines of the above-mentioned target regions and the contours of the above-mentioned target regions. In an alternative implementation manner, as Figure 3 shown, when the contours of the above-mentioned target regions are rectangles, determining at least four groups of corresponding point pairs based on the centerlines of the above-mentioned target regions and the contours of the above-mentioned target regions includes:

[0059] Step S302: Based on the above first center line, the contour of the target region in the above first image, and the contour of the target region in the above second image, determine the corresponding point pairs in the first direction.

[0060] Optionally, the above-mentioned based on the above first center line, the contour of the target region in the above first image, and the contour of the target region in the above second image may include: determining four intersection points of the above first center line with the contour of the target region in the above first image and the contour of the target region in the above second image; corresponding the above four intersection points in pairs to obtain the corresponding point pairs in the first direction.

[0061] Step S304: Based on the above second center line, the contour of the target region in the above first image, and the contour of the target region in the above third image, determine the corresponding point pairs in the second direction.

[0062] Optionally, the above-mentioned based on the above second center line, the contour of the target region in the above first image, and the contour of the target region in the above third image to determine the corresponding point pairs in the second direction may include: determining four intersection points of the above second center line with the contour of the target region in the above first image and the contour of the target region in the above second image; corresponding the above four intersection points in pairs to obtain the corresponding point pairs in the second direction.

[0063] It can be understood that when the target region is rectangular, since the target region in the above first image is aligned with the target region in the above second image in the first direction, the target region in the above first image is aligned with the target region in the above third image in the second direction, the above first center line is the center line of the target regions in the above first image and the above second image in the first direction, and the above second center line is the center line of the target regions in the above first image and the above third image in the second direction. Therefore, the 2 intersection points determined by the above first center line and the contour of the target region in the above first image, and the 2 intersection points determined by the above first center line and the contour of the target region in the above second image can directly determine 2 groups of corresponding point pairs; the 2 intersection points determined by the above second center line and the contour of the target region in the above first image, and the 2 intersection points determined by the above second center line and the contour of the target region in the above third image can also directly determine 2 groups of corresponding point pairs. In this way, compared with the prior art, the embodiment of the present disclosure omits the stereo matching process, that is, steps such as matching cost calculation and cost aggregation do not need to be performed in the process of determining the corresponding point pairs, reducing the calculation amount and having a low algorithm complexity.

[0064] Optionally, the first direction may be the horizontal direction of the image, and the second direction may be the vertical direction of the image. In a specific embodiment, the above-mentioned first center line is the center line in the horizontal direction of the target regions in the first image and the second image, and the above-mentioned second center line is the center line in the vertical direction of the target regions in the first image and the third image.

[0065] In this embodiment, based on the center lines of the above-mentioned target regions and the contours of the above-mentioned target regions, at least four groups of corresponding point pairs are determined, including:

[0066] Based on the above-mentioned first center line, the contour of the target region in the first image, and the contour of the target region in the second image, corresponding point pairs in the horizontal direction of the image are determined;

[0067] Based on the above-mentioned second center line, the contour of the target region in the first image, and the contour of the target region in the third image, corresponding point pairs in the vertical direction of the image are determined.

[0068] For example, when the target region is a rectangle, the above-mentioned first center line and the contour of the target region in the first image form 2 intersection points, which are the left intersection point 1 and the right intersection point 1 respectively. The above-mentioned first center line and the contour of the target region in the second image form 2 intersection points, which are the left intersection point 2 and the right intersection point 2 respectively. (Left intersection point 1, left intersection point 2) and (right intersection point 1, right intersection point 2) are respectively used as 2 groups of corresponding point pairs in the horizontal direction of the image. Similarly, the above-mentioned second center line and the contour of the target region in the first image form 2 intersection points, which are the upper intersection point 1 and the lower intersection point 1 respectively. The above-mentioned second center line and the contour of the target region in the third image form 2 intersection points, which are the upper intersection point 2 and the lower intersection point 2 respectively. (Upper intersection point 1, upper intersection point 2) and (lower intersection point 1, lower intersection point 2) are respectively used as 2 groups of corresponding point pairs in the vertical direction of the image.

[0069] In another alternative embodiment, as Figure 4 shown, the determination of the center lines of the target regions in the first image, the second image, and the third image includes:

[0070] Step S402, when the contours of the above-mentioned target regions are not rectangles, determine the first circumscribed rectangle, the second circumscribed rectangle, and the third circumscribed rectangle; wherein, the first circumscribed rectangle, the second circumscribed rectangle, and the third circumscribed rectangle are the circumscribed rectangles of the contours of the target regions in the first image, the second image, and the third image respectively. The first circumscribed rectangle and the second circumscribed rectangle are aligned in the first direction, and the first circumscribed rectangle and the third circumscribed rectangle are aligned in the second direction.

[0071] Step S404, determine the third center line and the fourth center line; wherein, the third center line is the center line of the first circumscribed rectangle and the second circumscribed rectangle in the first direction, and the fourth center line is the center line of the first circumscribed rectangle and the third circumscribed rectangle in the second direction.

[0072] It can be understood that when the contours of the target regions are not rectangles, to accurately obtain the center lines of the contours of the target regions, it is necessary to traverse each pixel of the contour, which increases the computational complexity. In this embodiment, the circumscribed rectangles of the contours are used to approximate the target regions, and then the third center line in the first direction and the fourth center line in the second direction are calculated. Thus, the third center line and the fourth center line can be approximately considered as the center lines of the target regions in the first direction and the second direction. By using this method, it is possible to avoid traversing the contours of the target regions, reduce the computational complexity, and make the real-time performance of the algorithm better.

[0073] The embodiments of the present disclosure do not limit the specific manner of determining at least four sets of corresponding point pairs based on the center lines of the target regions and the contours of the target regions. In an alternative embodiment, as Figure 5 shown, when the contours of the target regions are not rectangles, based on the center lines of the target regions and the contours of the target regions, determining at least four sets of corresponding point pairs includes:

[0074] Step S502, based on the third center line, the contour of the target region in the first image, and the contour of the target region in the second image, determine the corresponding point pairs in the first direction.

[0075] Optionally, the above-mentioned based on the third center line, the contour of the target region in the first image, and the contour of the target region in the second image may include: determining at least four intersection points of the third center line with the contour of the target region in the first image and the contour of the target region in the second image; pairing the at least four intersection points in pairs to obtain the corresponding point pairs in the first direction. It can be understood that when the contour of the target region is an irregular shape, the third center line and the contour of the target region in the first image and the contour of the target region in the second image may obtain at least four intersection points. For example, the number of intersection points may be 4, 6, 8,....

[0076] Step S504, based on the fourth center line, the contour of the target region in the first image, and the contour of the target region in the third image, determine the corresponding point pairs in the second direction.

[0077] Optionally, determining the corresponding point pairs in the second direction based on the fourth center line, the contour of the target region in the first image, and the contour of the target region in the third image may include: determining at least four intersection points of the fourth center line with the contour of the target region in the first image and the contour of the target region in the third image; and corresponding the at least four intersection points in pairs to obtain the corresponding point pairs in the second direction. It can be understood that when the contour of the target region is an irregular shape, the fourth center line and the contour of the target region in the first image and the contour of the target region in the third image may obtain at least four intersection points. For example, the number of intersection points may be 4, 6, 8,....

[0078] It can be understood that since the target region in the first image and the target region in the second image are aligned in the first direction, the target region in the first image and the target region in the third image are aligned in the second direction, the first circumscribed rectangle and the second circumscribed rectangle are aligned in the first direction, the first circumscribed rectangle and the third circumscribed rectangle are aligned in the second direction, the third center line is the center line of the first circumscribed rectangle and the second circumscribed rectangle in the first direction, and the fourth center line is the center line of the first circumscribed rectangle and the third circumscribed rectangle in the second direction. Therefore, at least two intersection points determined by the third center line and the contour of the target region in the first image and at least two intersection points determined by the third center line and the contour of the target region in the second image can directly determine at least two groups of corresponding point pairs; at least two intersection points determined by the fourth center line and the contour of the target region in the first image and at least two intersection points determined by the fourth center line and the contour of the target region in the third image can also be directly determined as at least two groups of corresponding point pairs. Thus, compared with the prior art, the embodiment of the present disclosure omits the stereo matching process, that is, steps such as matching cost calculation and cost aggregation are not required in the process of determining the corresponding point pairs, reducing the calculation amount and having a low algorithm complexity.

[0079] Optionally, the first direction may be the horizontal direction of the image, and the second direction may be the vertical direction of the image. In a specific embodiment, the third center line may be the center line of the first circumscribed rectangle and the second circumscribed rectangle in the horizontal direction of the image, and the fourth center line may be the center line of the first circumscribed rectangle and the third circumscribed rectangle in the vertical direction of the image. In this embodiment, determining at least four groups of corresponding point pairs based on the center lines of the target regions and the contours of the target regions includes:

[0080] Determining the corresponding point pairs in the horizontal direction of the image based on the third center line, the contour of the target region in the first image, and the contour of the target region in the second image;

[0081] Based on the above-mentioned fourth center line, the contour of the target area in the above-mentioned first image, and the contour of the target area in the above-mentioned third image, corresponding point pairs in the vertical direction of the image are determined.

[0082] Step S108: Based on the above-mentioned at least four groups of corresponding point pairs, the target pose is determined.

[0083] The embodiments of the present disclosure do not limit the specific manner of determining the target pose based on the above-mentioned at least four groups of corresponding point pairs. In a feasible embodiment, as Figure 6 shown, the determination of the target pose based on at least four groups of corresponding point pairs includes:

[0084] Step S602: Based on at least four groups of corresponding point pairs, at least four target points are determined; wherein, each of the above-mentioned target points is determined by a group of corresponding point pairs.

[0085] This embodiment does not limit the specific method of determining a target point by a group of corresponding point pairs, as long as the three-dimensional coordinates of the target point can be determined. Optionally, based on the calibration parameters of the camera, a group of corresponding point pairs can be converted into corresponding three-dimensional space coordinates; the three-dimensional coordinates of the target point are calculated by the triangulation method.

[0086] Step S604: Determine the centroid of the above-mentioned at least four target points.

[0087] This embodiment does not limit the specific manner of determining the centroid of the above-mentioned at least four target points. Optionally, the average values of the x coordinates, y coordinates, and z coordinates of the above-mentioned at least four target points can be calculated respectively as the three-dimensional coordinates (x, y, z) of the centroid. Compared with the prior art of first constructing a dense point cloud of an object and then calculating the centroid of the point cloud, this embodiment approximately replaces the center point of the target area with the centroid of the above-mentioned at least four target points, reducing the amount of data to be processed. It can be understood that the above-mentioned at least four target points include 4 boundary points in the positive and negative directions of the first direction and the positive and negative directions of the second direction of the contour of the target area. Since the first direction is perpendicular to the second direction, it can be considered that the centroid position of the above-mentioned 4 boundary points is close to the center point of the target area. Preferably, when the first direction is the horizontal direction of the image and the contour of the target area is a rectangle, the 4 target points determined are the midpoints of the 4 sides of the contour of the target area. Thus, the centroid coordinates determined based on the 4 target points are the same as the center point coordinates of the target area.

[0088] Step S606: Based on the above-mentioned at least four target points, the plane where they are located is fitted.

[0089] This embodiment does not limit the specific method of fitting the plane where the above at least four target points are located. Optionally, the above method of fitting the plane where the above at least four target points are located may include: using the least squares method to fit the plane where the above at least four target points are located based on the above at least four target points.

[0090] Assume that the expression of the plane equation is Bx + Cy + Dz + E = 0 (D≠0), and transform the above expression into a second expression:

[0091] Let Transform the above second expression into a third expression: z = a0x + a1y + a2;

[0092] Based on the above third expression, obtain its corresponding system of equations Ax = b; where, n is a positive integer and n≥4, (x1, y1, z1), (x2, y2, z2),..., (x n , y n , z n ) are the three-dimensional coordinates of the above at least four target points;

[0093] Solve the above system of equations based on the three-dimensional coordinates of the above at least four target points to obtain the optimal least squares solutions of a0, a1, and a2;

[0094] Based on the optimal least squares solutions of a0, a1, and a2, fit the plane where the above four target points are located.

[0095] Optionally, the above method of fitting the plane where the above at least four target points are located may further include: using the Lagrange multiplier method or the singular value decomposition method to fit the plane where the above at least four target points are located based on the above at least four target points.

[0096] Step S608, determine the above target pose based on the above centroid and the normal vector of the above plane.

[0097] This embodiment does not limit the specific form of the above target pose. In a specific embodiment, the above target pose may include 6 components. Optionally, the three-dimensional coordinates of the above centroid may be used as 3 components of the target pose, and the rotation amounts of the above normal vector relative to the coordinate axes of the camera coordinate system may be used as 3 components of the target pose. Optionally, the three-dimensional coordinates of the above centroid may be used as 3 components of the target pose, and the three-dimensional coordinates of the above normal vector may be used as 3 components of the target pose.

[0098] In a specific embodiment, the above method of determining the above target pose based on the above centroid and the normal vector of the above plane may further include: determining the above target pose based on the above centroid and the normal line of the above plane.

[0099] In summary, based on the centerlines of the target regions in the first image, the second image, and the third image and the contours of the above-mentioned target regions, the embodiments of the present disclosure can determine at least four groups of corresponding point pairs. Compared with the prior art, the stereo matching process is omitted, the amount of calculation is reduced, and the algorithm complexity is low. Secondly, based on the at least four groups of corresponding point pairs determined above, the present disclosure determines the target pose without constructing a dense point cloud of the object, reduces the amount of data to be processed, and improves the speed of pose estimation, so that it can be stably used outdoors.

[0100] The embodiments of the present disclosure also provide a target pose determination device 700 based on binocular vision. As Figure 7 shown, the above-mentioned device includes:

[0101] An acquisition module 701, configured to acquire a first image, a second image, and a third image; wherein, the first image, the second image, and the third image are images obtained by a first camera, a second camera, and a third camera respectively for photographing the same target, the target region in the first image is aligned with the target region in the second image in a first direction, the target region in the first image is aligned with the target region in the third image in a second direction, and the first direction is perpendicular to the second direction;

[0102] A centerline determination module 702, configured to determine the centerlines of the target regions in the first image, the second image, and the third image;

[0103] A corresponding point pair determination module 703, configured to determine at least four groups of corresponding point pairs based on the centerlines of the above-mentioned target regions and the contours of the above-mentioned target regions;

[0104] A pose determination module 704, configured to determine the target pose based on the at least four groups of corresponding point pairs.

[0105] In a possible implementation manner, the above-mentioned target pose determination device further includes a target exposure parameter determination module, configured to determine the target exposure parameters of the first camera, the second camera, or the third camera. The target exposure parameter determination module includes:

[0106] An acquisition sub-module, configured to acquire a plurality of reference images; wherein, the plurality of reference images are images obtained by the first camera, the second camera, or the third camera for photographing the same target under multiple groups of preset exposure parameters, the plurality of reference images include a first reference image, and the first reference image is an image obtained when the fill light of the camera is not turned on;

[0107] A fusion sub-module, configured to fuse the plurality of reference images to obtain a second reference image;

[0108] A grayscale value determination sub-module, configured to determine a first grayscale value and a second grayscale value, where the first grayscale value is the average grayscale value of the target area of the first reference image, and the second grayscale value is the average grayscale value of the target area of the second reference image; the target areas of the first reference image and the second reference image correspond to the same area of the target.

[0109] A target exposure parameter determination sub-module, configured to determine the target exposure parameter of the first camera, the second camera, or the third camera based on the first grayscale value and the second grayscale value.

[0110] In a possible implementation manner, the center line determination module is configured to: when the contours of the target areas are rectangles, determine a first center line and a second center line; where the first center line is the center line of the target areas in the first direction in the first image and the second image, and the second center line is the center line of the target areas in the second direction in the first image and the third image.

[0111] In this implementation manner, the corresponding point pair determination module is configured to: determine the corresponding point pairs in the first direction based on the first center line, the contour of the target area in the first image, and the contour of the target area in the second image; determine the corresponding point pairs in the second direction based on the second center line, the contour of the target area in the first image, and the contour of the target area in the third image.

[0112] In another possible implementation manner, the center line determination module is configured to: when the contours of the target areas are not rectangles, determine a first circumscribed rectangle, a second circumscribed rectangle, and a third circumscribed rectangle; where the first circumscribed rectangle, the second circumscribed rectangle, and the third circumscribed rectangle are the circumscribed rectangles of the contours of the target areas in the first image, the second image, and the third image respectively, the first circumscribed rectangle is aligned with the second circumscribed rectangle in the first direction, and the first circumscribed rectangle is aligned with the third circumscribed rectangle in the second direction; determine a third center line and a fourth center line; where the third center line is the center line of the first circumscribed rectangle and the second circumscribed rectangle in the first direction, and the fourth center line is the center line of the first circumscribed rectangle and the third circumscribed rectangle in the second direction.

[0113] In this implementation manner, the corresponding point pair determination module is configured to: determine the corresponding point pairs in the first direction based on the third center line, the contour of the target area in the first image, and the contour of the target area in the second image; determine the corresponding point pairs in the second direction based on the fourth center line, the contour of the target area in the first image, and the contour of the target area in the third image.

[0114] In a possible implementation, the above pose determination module is configured to: determine at least four target points based on at least four sets of corresponding points; wherein each of the above target points is determined by a set of corresponding points; determine the centroid of the above at least four target points; fit a plane where they are located based on the above at least four target points; and determine the above target pose based on the above centroid and the normal vector of the above plane.

[0115] In summary, the embodiments of the present disclosure provide a target pose determination device based on binocular vision. The corresponding method embodiments above are all based on the same inventive concept. For details, please refer to the method embodiments and will not be elaborated here.

[0116] The embodiments of the present disclosure also provide an electronic device, including: at least one processor; a memory for storing instructions executable by the above at least one processor; wherein the above at least one processor is configured to execute the above instructions to implement the above method disclosed in the embodiments of the present disclosure.

[0117] Figure 8 It is a schematic structural diagram of an electronic device provided for an exemplary embodiment of the present disclosure. As Figure 8 shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can execute the corresponding steps in the above method disclosed in the embodiments of the present disclosure.

[0118] The above processor 1801 can also be referred to as a central processing unit (CPU). It can be an integrated circuit chip with signal processing capabilities. Each step in the above method disclosed in the embodiments of the present disclosure can be completed by the integrated logic circuit in the hardware of the processor 1801 or the instructions in software form. The above processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module can be located in the memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, and other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and combines its hardware to complete the steps of the above method.

[0119] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, a program constituting the software can be installed from a storage medium or a network into a computer system having a dedicated hardware structure, such as Figure 9 the computer system 1900 shown. When various programs are installed in the computer system, it can execute various functions, including, for example, the functions described above. Figure 9 The block diagram of the computer system provided by an exemplary embodiment of the present disclosure.

[0120] The computer system 1900 is intended to represent various forms of digital electronic computer devices, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic devices can also represent various forms of mobile devices, such as, for example, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0121] As Figure 9 shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0122] Multiple components in computer system 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. The input unit 1906 can be any type of device capable of inputting information into computer system 1900. The input unit 1906 can receive input digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 1907 can be any type of device capable of presenting information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 can include but is not limited to magnetic disks and optical discs. The communication unit 1909 allows computer system 1900 to exchange information / data with other devices via a network such as the Internet, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0123] The computing unit 1901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 executes the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed onto computer system 1900 via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 can be configured to execute the above methods disclosed in the embodiments of the present disclosure by any other suitable means (e.g., by means of firmware).

[0124] The embodiments of the present disclosure also provide a computer-readable storage medium, wherein when the instructions in the above computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above methods disclosed in the embodiments of the present disclosure.

[0125] The computer-readable storage medium in the embodiments of the present disclosure may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The above computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination of the foregoing. More specifically, the above computer-readable storage medium may include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0126] The above computer-readable medium may be included in the above electronic device; or may exist separately without being assembled into the electronic device.

[0127] The embodiments of the present disclosure also provide a computer program product, including a computer program, wherein the above computer program, when executed by a processor, implements the above method disclosed in the embodiments of the present disclosure.

[0128] In the embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The above programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0130] The modules, components, or units described in the embodiments of the present disclosure can be implemented in software or in hardware. Among them, the names of the modules, components, or units do not, in some cases, constitute a limitation on the modules, components, or units themselves.

[0131] The functions described above can be at least partially performed by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0132] The above description is only some embodiments of the present disclosure and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by replacing the above features with other technical features (but not limited to) having similar functions disclosed in the present disclosure.

[0133] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art should understand that the above examples are only for illustration and not for limiting the scope of the present disclosure. Those skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for determining the target pose based on binocular vision, characterized in that, Including: Obtain a first image, a second image, and a third image; wherein, the first image, the second image, and the third image are images obtained by a first camera, a second camera, and a third camera respectively for the same target, the target regions in the first image and the second image are aligned in a first direction, the target regions in the first image and the third image are aligned in a second direction, and the first direction is perpendicular to the second direction; Determine the centerlines of the target regions in the first image, the second image, and the third image; Based on the centerlines of the target regions and the contours of the target regions, determine at least four sets of corresponding point pairs; Based on the at least four sets of corresponding point pairs, determine the target pose.

2. The method according to claim 1, characterized in that, Before obtaining the first image, the second image, and the third image, it further includes: Obtain a plurality of reference images; wherein, the plurality of reference images are images obtained by the first camera, the second camera, or the third camera for the same target under multiple sets of preset exposure parameters, the plurality of reference images include a first reference image, and the first reference image is an image obtained when the fill light of the camera is not turned on; Fuse the plurality of reference images to obtain a second reference image; Determine a first gray value and a second gray value, where the first gray value is the average gray value of the target region of the first reference image, and the second gray value is the average gray value of the target region of the second reference image; the target regions of the first reference image and the second reference image correspond to the same region of the target; Based on the first gray value and the second gray value, determine the target exposure parameters of the first camera, the second camera, or the third camera.

3. The method according to claim 1, characterized in that, The determination of the centerlines of the target regions in the first image, the second image, and the third image includes: When the contours of the target regions are rectangles, determine a first centerline and a second centerline; wherein, the first centerline is the centerline of the target regions in the first image and the second image in the first direction, and the second centerline is the centerline of the target regions in the first image and the third image in the second direction.

4. The method according to claim 3, wherein The determination of at least four sets of corresponding point pairs based on the centerlines of the target regions and the contours of the target regions includes: Based on the first centerline, the contour of the target region in the first image, and the contour of the target region in the second image, determine the corresponding point pairs in the first direction; Based on the second centerline, the contour of the target region in the first image, and the contour of the target region in the third image, determine the corresponding point pairs in the second direction.

5. The method according to claim 1, characterized in that, The determination of the centerlines of the target regions in the first image, the second image, and the third image includes: When the contours of the target regions are not rectangles, determine a first circumscribed rectangle, a second circumscribed rectangle, and a third circumscribed rectangle; wherein, the first circumscribed rectangle, the second circumscribed rectangle, and the third circumscribed rectangle are respectively the circumscribed rectangles of the contours of the target regions in the first image, the second image, and the third image, the first circumscribed rectangle is aligned with the second circumscribed rectangle in a first direction, and the first circumscribed rectangle is aligned with the third circumscribed rectangle in a second direction; Determine a third center line and a fourth center line; wherein, the third center line is the center line of the first circumscribed rectangle and the second circumscribed rectangle in the first direction, and the fourth center line is the center line of the first circumscribed rectangle and the third circumscribed rectangle in the second direction.

6. The method according to claim 5, wherein Based on the center lines of the target regions and the contours of the target regions, determine at least four groups of corresponding point pairs, including: Based on the third center line, the contour of the target region in the first image, and the contour of the target region in the second image, determine the corresponding point pairs in the first direction; Based on the fourth center line, the contour of the target region in the first image, and the contour of the target region in the third image, determine the corresponding point pairs in the second direction.

7. The method according to claim 1, characterized in that Based on the at least four groups of corresponding point pairs, determine the target pose, including: Based on the at least four groups of corresponding point pairs, determine at least four target points; wherein, each target point is determined by a group of corresponding point pairs; Determine the centroid of the at least four target points; Based on the at least four target points, fit the plane where they are located; Based on the centroid and the normal vector of the plane, determine the target pose.

8. An object pose determination device based on binocular vision, characterized in that Include: An acquisition module, configured to acquire a first image, a second image, and a third image; wherein, the first image, the second image, and the third image are respectively images captured by a first camera, a second camera, and a third camera for the same target, the target region in the first image is aligned with the target region in the second image in a first direction, the target region in the first image is aligned with the target region in the third image in a second direction, and the first direction is perpendicular to the second direction; A center line determination module, configured to determine the center lines of the target regions in the first image, the second image, and the third image; A corresponding point pair determination module, configured to determine at least four groups of corresponding point pairs based on the center lines of the target regions and the contours of the target regions; A pose determination module, configured to determine the target pose based on the at least four groups of corresponding point pairs.

9. An electronic device, characterized in that, Include: At least one processor; A memory for storing instructions executable by the at least one processor; Wherein, the at least one processor is configured to execute the instructions to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1-7 is implemented.

Images