Pose solving processing method, device and equipment based on multi-vision sensor

Abstract

The application discloses a pose solution processing method, device and equipment based on a multi-vision sensor. The method comprises the following steps: detecting a marker by using the multi-vision sensor in a current frame; triggering a current environment adjustment processing procedure when only no more than one vision sensor in the multi-vision sensor detects the marker; when at least two eyes in the multi-vision sensor detect the marker, judging whether a pose solution result of the at least two eyes relative to the marker meets a preset accuracy requirement according to a relative pose parameter between the at least two eyes; and taking the pose solution result of the at least two eyes relative to the marker as a pose solution result of the current frame when the preset accuracy requirement is met. According to the technical scheme, the problem that a camera in related technologies is easily affected by environmental conditions when detecting a marker can be solved, so that the recognition rate of the marker is improved, and the pose solution accuracy is also improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular, to a pose solving method and device based on a multi-view vision sensor and an equipment. BACKGROUND

[0002] In the field of intelligent robots, initializing the world system origin point of the autonomous navigation robot is an indispensable part of the robot positioning system.

[0003] In recent years, the technology of positioning the world system origin point based on visual markers has been increasingly favored due to its low cost, fast positioning, and simple use. The technology based on visual markers generally uses a camera to detect markers (such as a planar marker plate, etc.) and solve the pose (position and azimuth angle) between the camera and the markers, thereby realizing the initialization positioning process of the robot.

[0004] The existing technical solutions based on visual sensors (such as cameras, etc.) generally have the following combinations: based on a monocular camera and a single marker; based on a single camera and multiple markers; based on multiple cameras (more than or equal to two cameras, such as multi-view cameras) and a single marker; and based on multiple cameras and multiple markers.

[0005] However, the scheme based on visual sensors also has the disadvantage of being easily affected by the external environment. For example, in a weak light or dark environment, the camera can hardly or cannot recognize the markers, and similar problems also exist in very strong sunlight. However, when the camera is partially or completely blocked, the markers cannot be detected.

[0006] Due to the influence of environmental conditions, the use of visual markers is greatly limited. Therefore, how to reduce the influence of the environment on the technology of positioning the world origin point based on camera detection of markers to improve the working efficiency of the robot is a problem to be solved at present. SUMMARY

[0007] The main purpose of the present application is to disclose a pose solving method and device based on a multi-view vision sensor, to at least solve the problems in the related art that the technology of detecting markers based on cameras is easily affected by environmental conditions, resulting in a low recognition rate and greatly limiting the use of visual marker detection.

[0008] According to one aspect of the present application, a pose solving method based on a multi-view vision sensor is provided.

[0009] The pose solving processing method based on the multi-vision sensor according to the present application comprises: detecting a marker in a current frame by using a multi-vision sensor; triggering an adjustment processing procedure of a current environment when no more than one vision sensor in the multi-vision sensor detects the marker; determining whether a pose solving result of at least two eyes relative to the marker meets a preset accuracy requirement according to a relative pose parameter between the at least two eyes when the at least two eyes in the multi-vision sensor detect the marker; and taking the pose solving result of the at least two eyes relative to the marker as a pose solving result of the current frame when the pose solving result of the at least two eyes relative to the marker meets the preset accuracy requirement.

[0010] According to another aspect of the present application, a pose solving processing device based on a multi-vision sensor is provided.

[0011] The pose solving processing device based on the multi-vision sensor according to the present application comprises: a detection module configured to detect a marker in a current frame by using a multi-vision sensor; an adjustment module configured to trigger an adjustment processing procedure of a current environment when no more than one vision sensor in the multi-vision sensor detects the marker; a determination module configured to determine whether a pose solving result of at least two eyes relative to the marker meets a preset accuracy requirement according to a relative pose parameter between the at least two eyes when the at least two eyes in the multi-vision sensor detect the marker; and a saving module configured to take the pose solving result of the at least two eyes relative to the marker as a pose solving result of the current frame when the pose solving result of the at least two eyes relative to the marker meets the preset accuracy requirement.

[0012] According to still another aspect of the present application, a pose solving processing apparatus based on a multi-vision sensor is provided.

[0013] The pose solving processing device based on the multi-vision sensor according to the present application comprises: a multi-vision sensor, a photosensitive sensor, a light supplementing device, a memory and a processor, wherein the multi-vision sensor is configured to detect a marker; the photosensitive sensor is configured to detect a current light condition; the light supplementing device is configured to supplement light when the light is insufficient; the memory is configured to store computer execution instructions; and the processor is configured to execute the computer execution instructions stored in the memory, so that the robot executes the pose solving processing method.

[0014] According to the present application, when using a multi-vision sensor to detect a marker, in the case that only no more than one vision sensor (only 1 vision sensor or 0 vision sensor) in the multi-vision sensor detects the marker, a current environment adjustment processing procedure is triggered to be executed; by setting the adjustment processing procedure, the problem that in the related art, a marker is detected based on a camera, is easily affected by environmental conditions, and thus recognition rate is reduced, can be solved. Moreover, when at least two (2 or more vision sensors) in the multi-vision sensor detect the marker, whether a pose solution result of the at least two relative to the marker meets a preset accuracy requirement can be determined according to a relative pose parameter between the at least two, and when the pose solution result of the at least two relative to the marker meets the preset accuracy requirement, the pose solution result of the at least two relative to the marker is taken as a pose solution result of a current frame. Through the accuracy determination processing scheme, the pose solution accuracy is improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of a multi-vision sensor-based pose solution processing method according to an embodiment of the present application;

[0016] Figure 2 is a marker detection flowchart based on a binocular vision sensor according to a preferred embodiment of the present application;

[0017] Figure 3 is a pose solution accuracy determination processing flowchart based on a binocular vision sensor according to a preferred embodiment of the present application;

[0018] Figure 4 is a structural block diagram of a multi-vision sensor-based pose solution processing apparatus according to an embodiment of the present application;

[0019] Figure 5 is a structural block diagram of a multi-vision sensor-based pose solution processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The specific implementation modes of the present application will be described in detail below with reference to the accompanying drawings.

[0021] According to an embodiment of the present application, a multi-vision sensor-based pose solution processing method is provided.

[0022] Figure 1 is a flowchart of a multi-vision sensor-based pose solution processing method according to an embodiment of the present application. As shown in Figure 1 the multi-vision sensor-based pose solution processing method includes:

[0023] Step S101: detecting a marker using a multi-view visual sensor in a current frame;

[0024] Step S102: in a case where no more than one visual sensor in the multi-view visual sensor detects the marker, triggering execution of an adjustment processing procedure of a current environment;

[0025] Step S103: when at least two views in the multi-view visual sensor detect the marker, determining whether a pose solution result of the at least two views relative to the marker satisfies a preset accuracy requirement according to a relative pose parameter between the at least two views.

[0026] Step S104: in a case where the pose solution result of the at least two views relative to the marker satisfies the preset accuracy requirement, taking the pose solution result of the at least two views relative to the marker as a pose solution result of the current frame.

[0027] In the related art, the marker detection technology based on a camera is susceptible to environmental conditions, which leads to a low recognition rate and greatly limits the use scenarios of the marker detection based on vision. The marker detection technology based on a multi-view visual sensor (for example, a multi-view camera, a multi-view visual module, etc.) is used. Figure 1 As shown in the marker detection technology based on a multi-view visual sensor (for example, a multi-view camera, a multi-view visual module, etc.), when the marker is detected using the multi-view visual sensor, in a case where no more than one visual sensor (only one visual sensor or no visual sensor) in the multi-view visual sensor detects the marker, an adjustment processing procedure of a current environment is triggered to be executed. By setting the adjustment processing procedure, the problem that the marker detection based on a camera in the related art is susceptible to environmental conditions and leads to a low recognition rate can be solved. When at least two views (two or more visual sensors) in the multi-view visual sensor detect the marker, whether a pose solution result of the at least two views relative to the marker satisfies a preset accuracy requirement can be determined according to a relative pose parameter between the at least two views. In a case where the pose solution result of the at least two views relative to the marker satisfies the preset accuracy requirement, the pose solution result of the at least two views relative to the marker is taken as a pose solution result of a current frame. By using the accuracy determination processing scheme, the pose solution accuracy is improved.

[0028] Of course, the marker can be an object containing various feature markers such as natural features and fixed features, for example, a planar marker plate, etc. The marker on the marker can be an apriltag or other types of markers such as ARTag, ARToolkit, aruco, etc.

[0029] In the preferred implementation, a plurality of (2 or more) visual sensors are arranged at intervals to form a multi-view visual sensor device for detecting a marker (e.g., an apriltag marker board, etc.). When at least two of the multi-view visual sensors detect the marker, a determination is made as to whether the pose solution of the at least two views relative to the marker meets a predetermined accuracy requirement based on the relative pose parameters between the at least two views. When the pose solution of the at least two views relative to the marker meets the predetermined accuracy requirement, the pose solution of the at least two views relative to the marker is taken as the pose solution of the current frame. If no more than one visual sensor (0 or 1 visual sensor) detects the marker, a current environment adjustment process is triggered. When at least two of the multi-view visual sensors detect the marker, a determination is made as to whether the pose solution of the at least two views relative to the marker meets a predetermined accuracy requirement based on the relative pose parameters between the at least two views. When the pose solution of the at least two views relative to the marker meets the predetermined accuracy requirement, the pose solution of the at least two views relative to the marker is taken as the pose solution of the current frame.

[0030] Preferably, the current environment adjustment process can further include the following processing: using a photosensitive sensor to detect the current light condition; when the light is overexposed, adjusting the exposure time and / or gain of the visual sensor, and then continuing to detect the marker, and repeating this step until at least two of the multi-view visual sensors detect the marker; when the light is insufficient, using a light supplement device to adjust the light supplement intensity, and then continuing to detect the marker, and repeating this step until at least two of the multi-view visual sensors detect the marker.

[0031] Preferably, before using the photosensitive sensor to detect the current light condition, the following processing can also be included: equalizing the histogram of the marker, continuing to detect the marker, and repeating this step to determine whether at least two of the multi-view visual sensors detect the marker.

[0032] The following is an example of a binocular visual sensor. In the preferred implementation, a binocular visual sensor is used to detect a marker. If both binoculars detect the marker, the pose solution of the two binoculars relative to the marker is saved. If one binocular detects the marker, the pose solution of the binocular relative to the marker is saved. If the other binocular does not detect the marker, a current environment adjustment process is triggered. Alternatively, if neither binocular detects the marker, the visual sensor that does not detect the marker also needs to trigger the current environment adjustment process.

[0033] Optionally, the marker can be detected after histogram equalization of the marker image, and this step is executed in a loop to determine whether the visual sensor that has not detected the marker before detects the marker, and if the marker is detected, the pose solution of the visual sensor relative to the marker is saved.

[0034] If the marker cannot be detected by the visual sensor that has not detected the marker before through histogram equalization of the marker image, the current light condition is detected using the photosensitive sensor.

[0035] When the light is too bright (e.g., when a planar marker board is detected, the planar marker board is too bright in the image and it is difficult to detect), the exposure time and / or gain of the visual sensor are adjusted, e.g., the exposure time is reduced to reduce the number of photons on the photosensitive plane, and the brightness of the marker image is reduced, and then the result is reported to the code scanning application module, which continues to attempt to scan the code, i.e., continues to detect the marker, and this step is executed in a loop until the marker is detected by the visual sensor that has not detected the marker before. When the light is insufficient, a light supplement device (e.g., an infrared (IR) light supplement lamp) can be used to adjust the light supplement intensity, and then the marker is continuously detected, and this step is executed in a loop until the marker is detected by both eyes. The light supplement device can adjust the light supplement intensity using the following strategy: when the value of the photosensitive sensor is less than a predetermined threshold, i.e., the light is insufficient, the light supplement lamp brightness value range is set to [0, a], and a set of multiple brightness element values is determined from the range according to the insufficient light environment, and when a = 100, the set is {0, 20, 40, 60, 80, 90, 100}. Then, the brightness of the light supplement lamp is switched using a binary search strategy, e.g., the brightness of the light supplement lamp is first set to the middle value 60, and if it is not successful, the left or right is selected for searching, e.g., the right is selected for searching, and the values 80, 90, and 100 are selected, and then the brightness of the light supplement lamp is set to the middle value 90, and if it is not successful, the left is selected for searching, and the values 0, 20, and 40 are selected, and then the brightness of the light supplement lamp is set to the middle value 20, and if it is not successful, the left is selected for searching, and so on until it is successful.

[0036] Preferably, when the light is too bright, it is determined that the marker cannot be detected by at least two of the multiple visual sensors through adjustment of the exposure time and / or gain of the visual sensor, and the histogram of the marker is equalized, and the marker is continuously detected, and this step is executed in a loop until the marker is detected by at least two of the multiple visual sensors.

[0037] Preferably, when the light is insufficient, and it is determined that the above-mentioned multi-vision sensor cannot detect the above-mentioned marker by at least two eyes through adjusting the light compensation intensity, the exposure time and / or gain of the vision sensor is adjusted, and then the above-mentioned marker is continuously detected, and the step is repeatedly executed.

[0038] In the preferred implementation process, taking the binocular vision sensor as an example, when the light is insufficient, and it is determined that the above-mentioned multi-vision sensor cannot detect the above-mentioned marker by both eyes through adjusting the light compensation intensity, it is possible that one eye (left eye or right eye) of the binocular vision sensor is always in overexposure (overexposure caused by turning on the light compensation lamp) under the light compensation adjustment strategy, so the gain (or exposure time) can be adjusted to further attempt to scan the code.

[0039] Preferably, the above-mentioned steps are repeatedly executed, and when it is determined that the above-mentioned multi-vision sensor cannot detect the above-mentioned marker by at least two eyes through adjusting the exposure time and / or gain of the vision sensor, the histogram of the above-mentioned marker is equalized, and then the above-mentioned marker is continuously detected, and the step is repeatedly executed until the above-mentioned multi-vision sensor detects the above-mentioned marker by at least two eyes. It should be noted that if it is determined that the above-mentioned multi-vision sensor cannot detect the above-mentioned marker by at least two eyes through equalizing the histogram of the above-mentioned marker, the adjustment process of the current environment is determined to be ended.

[0040] In the specific implementation process, the following processing mode is further included: in the case that only one vision sensor of the above-mentioned multi-vision sensor detects the above-mentioned marker, the monocular pose solution result of the vision sensor relative to the above-mentioned marker is saved; when it is determined that the above-mentioned multi-vision sensor cannot detect the above-mentioned marker by at least two eyes through executing the above-mentioned adjustment process of the current environment, the saved monocular pose solution result is used as the pose solution result of the current frame; after executing the pose solution processing of multiple frames, when the pose solution result of the above-mentioned at least two eyes relative to the above-mentioned marker does not appear in each of the saved pose solution results of the frames, the second mean value of the monocular pose solution results of the at least two frames closest to the current frame is calculated, and the above-mentioned second mean value is used as the final pose solution result. That is, in a special case, for a predetermined number of frames (for example, 100 frames, 200 frames, etc.), if there is no frame in which the above-mentioned multi-vision sensor detects the above-mentioned marker by at least two eyes, only the monocular pose solution result can be used as the final pose solution result in order to improve the recognition rate. In order to improve the accuracy and ensure the stability, the mean value of the monocular pose solution results of the at least two frames closest to the current frame can be calculated by using the multi-frame fusion mode, and the mean value is used as the final pose solution result.

[0041] The above-mentioned preferred implementation process is further described below in combination with the examples of Figure 2 .

[0042] Figure 2 is a flowchart of a marker detection process based on a binocular vision sensor according to a preferred embodiment of the present application. As shown in Figure 2 , the marker (taking a planar marker board as an example) detection process mainly includes:

[0043] Step S201: detecting a planar marker board using a binocular vision sensor in a current frame;

[0044] Step S202: when only the left eye of the binocular vision sensor detects the planar marker board, saving the pose solution result corresponding to the left eye; and triggering execution of an adjustment processing flow of the current environment.

[0045] Step S203: performing histogram equalization processing on the planar marker board, continuing to detect the planar marker board, and repeatedly executing this step to determine whether the right eye detects the planar marker board; if yes, executing flow S211; and if no, executing step S204.

[0046] Step S204: in a case where it is determined that the right eye cannot detect the planar marker board by executing step S203, detecting a current light condition using a photosensitive sensor; when the light is overexposed, executing step S205; and when the light is insufficient, executing step S208.

[0047] Step S205: when the light is overexposed, adjusting the exposure time and / or gain of the right eye (for example, reducing the exposure time, etc.), and then reporting to a code scanning application, continuing to detect the planar marker board, and repeatedly executing this step to determine whether the right eye detects the planar marker board; if yes, executing step S211; and if no, executing step S206.

[0048] Step S206: when it is determined that the at least two eyes of the binocular vision sensor cannot detect the planar marker board by adjusting the exposure time and / or gain of the vision sensor that does not detect the planar marker board, performing histogram equalization processing on the planar marker board, continuing to detect the planar marker board, and repeatedly executing this step to determine whether the right eye detects the planar marker board; if yes, executing step S211; and if no, determining that the right eye detection process fails, and executing step S207.

[0049] Step S207: in a case where it is determined that the left and right eyes cannot detect the planar marker board by executing the adjustment processing flow of the current environment, saving the saved pose solution result of the left eye as a pose solution result of the current frame.

[0050] Step S208: When the light is insufficient, the light supplement device is used to adjust the light supplement intensity, and then the planar marker board is continuously detected. The step is executed in a loop to determine whether the right eye detects the planar marker board. If yes, step S211 is executed. If no, step S209 is executed.

[0051] Step S209: When the light is insufficient, it is determined that the light supplement device cannot be used to adjust the light supplement intensity to enable at least two eyes of the multi-eye vision sensor to detect the planar marker board. The exposure time and / or gain are adjusted, and then the planar marker board is continuously detected. The step is executed in a loop to determine whether the right eye detects the planar marker board. If yes, step S211 is executed. If no, step S210 is executed.

[0052] Step S210: When it is determined that the exposure time and / or gain of the vision sensor (for example, a camera) cannot be adjusted to enable the right eye to detect the planar marker board, the histogram of the planar marker board is equalized, and then the planar marker board is continuously detected. The step is executed in a loop to determine whether the right eye detects the planar marker board. If yes, step S211 is executed. If no, it is determined that the right eye detection process fails, and step S207 is executed.

[0053] Step S211: The detection process ends, and the pose solution result corresponding to the right eye is saved.

[0054] Preferably, in step 103, determining whether the pose solution result of the at least two eyes relative to the marker satisfies the preset accuracy requirement according to the relative pose parameters between the at least two eyes can further include: for each vision sensor in the at least two eyes, a pose solution result corresponding to the vision sensor is obtained, where each vision sensor corresponds to two solution matrices; and according to the relative pose parameters between two vision sensors in the at least two eyes and the pose solutions of the two vision sensors relative to the marker, it is determined whether the pose solution result of the two vision sensors relative to the marker satisfies the preset accuracy requirement.

[0055] Preferably, according to the relative pose parameters between the two vision sensors and the pose solutions of the two vision sensors relative to the marker, it is determined whether the pose solution result of the two vision sensors relative to the marker satisfies the preset accuracy requirement can further include:

[0056] The relative pose parameters between the two vision sensors are solved by the following formula: where x is a set {1, 2}, T 1x represents two pose solutions corresponding to a first vision sensor in the two vision sensors, T 2xrepresent two pose solutions corresponding to the second vision sensor of the two vision sensors, and the represent relative pose parameters between the two vision sensors; and compare the calculated relative pose parameters with the pre-determined relative pose parameters between the two vision sensors to determine correct pose solutions corresponding to the two vision sensors respectively; and determine whether the pose solutions of the two vision sensors satisfy a preset accuracy requirement according to errors between the correct pose solutions and the pre-determined relative pose parameters between the two vision sensors.

[0057] In the preferred implementation, the pose solution between the vision sensor (e.g., a camera) and the marker can be solved by using a PNP (Perspective-n-Point) algorithm to solve a 3D (three-dimensional) to 2D (two-dimensional) perspective projection method. However, when the vision sensor is far away from the marker, the marker projected onto the vision sensor plane is very small, resulting in a weakened perspective projection model, so that the solved rotation has two solutions (i.e., two rotation solutions). In addition, when the marker projected onto the vision sensor plane is too small, the influence of image noise is more obvious, which also increases the two solutions of the rotation. To solve this problem, a more robust IPPE (Infinitesimal Plane-based Pose Estimation) algorithm can be used, which can solve two solutions, but cannot distinguish which solution is correct. The correct solution can be found by using the advantages of binocular vision sensors. The following takes binocular vision sensors as an example to illustrate:

[0058] The pose is solved by using the IPPE method for the left and right eyes respectively, and then the pose solution of the IPPE is used as an initial solution to further optimize by using a Levenberg-Marquardt (LM) optimization algorithm method, wherein T 11 = [R 11 , t 11 ], T 12 = [R 12 , t 12 ] represent two optimized pose solutions of the left eye, and T 21 = [R 21 , t 21 ], T 22 = [R 22 , t 22 ] represent two optimized poses of the right eye. Since the pose between the binoculars is known, the correct pose solution of the left and right eyes can be found by using the following formula Wherein, x is a set {1,2}. That is, four groups of results can be obtained by combination, and the two pose solutions in the group closest to the known pose T between the two eyes are taken as the correct pose solutions corresponding to the two eyes respectively. And according to the correct pose solutions corresponding to the two eyes, the distance between the two eyes is calculated. And the error between the known pose between the two eyes and the calculated pose is determined whether it meets the preset accuracy requirement. If the error is greater than the preset threshold, the preset requirement is not met. If the error is less than or equal to the preset threshold, the preset accuracy requirement is met.

[0059] For example, for a predetermined number of frames (for example, 100 frames, 200 frames, etc.), if a plurality of frames at least two eyes detect the above-mentioned markers, and the pose solution of the above-mentioned at least two eyes relative to the markers meets the preset accuracy requirement, in order to improve the accuracy and ensure the stability, the multi-frame fusion method can be used to calculate the mean value of the pose solution of the at least two frames closest to the current, and the mean value is taken as the final pose solution.

[0060] Preferably, after determining whether the pose solution of the above-mentioned at least two eyes relative to the markers meets the preset accuracy requirement, the following processing can also be included: when the pose solution of the above-mentioned at least two eyes relative to the markers does not meet the above-mentioned preset accuracy requirement, the region of interest of the above-mentioned markers is selected, and the brightness equalization processing is performed on the above-mentioned region of interest; continue to determine whether the pose solution of the above-mentioned at least two eyes relative to the markers meets the preset accuracy requirement, and when the pose solution of the above-mentioned at least two eyes relative to the markers meets the above-mentioned preset accuracy requirement, save the pose solution of the above-mentioned at least two eyes relative to the markers.

[0061] Preferably, after continuing to determine whether the pose solution of the above-mentioned at least two eyes relative to the markers meets the preset accuracy requirement, the following processing can also be included: when the pose solution of the above-mentioned at least two eyes relative to the markers does not meet the above-mentioned preset accuracy requirement, the current light condition is detected using a photosensitive sensor; when the light is overexposed, the exposure time and / or gain of the visual sensor that does not detect the above-mentioned markers are adjusted, and then the above-mentioned markers are continued to be detected, and the step is executed in a loop until the above-mentioned markers are detected by at least two of the multi-eye visual sensors; when the light is insufficient, the light supplementing device is used to adjust the light supplementing intensity, and then the above-mentioned markers are continued to be detected, and the step is executed in a loop until the above-mentioned markers are detected by at least two of the multi-eye visual sensors.

[0062] In the preferred implementation process, when the light is overexposed, and it is determined that the at least two eyes of the multi-view vision sensor cannot detect the marker by adjusting the exposure time and / or gain of the vision sensor, the histogram of the marker is equalized, the marker is continuously detected, and the step is repeatedly executed until the at least two eyes of the multi-view vision sensor detect the marker.

[0063] In the preferred implementation process, when the light is insufficient, and it is determined that the at least two eyes of the multi-view vision sensor cannot detect the marker by adjusting the light supplement intensity using the light supplement device, the exposure time and / or gain of the vision sensor is adjusted, and then the marker is continuously detected, and the step is repeatedly executed; when it is determined that the at least two eyes of the multi-view vision sensor cannot detect the marker by adjusting the exposure time and / or gain of the vision sensor, the histogram of the marker is equalized, and then the marker is continuously detected, and the step is repeatedly executed until the at least two eyes of the multi-view vision sensor detect the marker.

[0064] Preferably, the pose solving processing method can be repeatedly executed until the pose solving results of the at least two eyes relative to the marker in multiple frames are obtained; for each of the at least two frames closest to the current frame in the multiple frames, the pose solving result of the frame is converted to the same coordinate system, and the mean value of the converted pose solving results is obtained; the first mean value is obtained according to the mean value of the pose solving result of each frame in the at least two frames, and the first mean value is taken as the final pose solving result.

[0065] For example, after the pose solving results of the left and right eyes relative to the marker in multiple frames (for example, 10 frames) are obtained, the closest 3 frames can be selected, the pose solving result of the right eye relative to the marker in each of the 3 frames is converted to the same coordinate system of the left eye through the relative pose parameters between the left and right eyes. Then, the mean value of the pose solving result of the left and right eyes relative to the marker in each frame is obtained, and then the mean value of the corresponding mean values of the 3 frames is obtained, that is, the first mean value, which is taken as the final pose solving result.

[0066] The above preferred implementation process is further described below in combination with the examples of Figure 3 .

[0067] Figure 3 is a flowchart of the pose solving accuracy judgment processing according to the preferred embodiment of the present application; as shown in Figure 3 , the pose solving accuracy judgment processing mainly includes:

[0068] Step S301; obtaining the pose solving result of the left and right eyes of the current frame relative to the marker (taking a planar marker board as an example for illustration), wherein, due to the binoculars being far away from the marker board, the marker board projected onto the binocular plane is small, resulting in the projection model being weakened into a perspective-like projection model, so that the solved rotation has two solutions (there are two rotation solutions), in addition, when the marker board projected onto the binocular plane is too small, the influence of image noise is more obvious, and the two solutions of the rotation are also increased, so that the left and right eyes correspond to two pose solutions respectively; for example, T 11 = [R 11 ,t 11 ], T 12 = [R 12 ,t 12 ] represents two optimized poses of the left eye, T 21 = [R 21 ,t 21 ], T 22 = [R 22 ,t 22 ] represents two optimized poses of the right eye. Wherein, R represents a rotation matrix, and t represents a translation matrix.

[0069] Step S302; the relative pose parameters between the binoculars are solved by the following formula: Wherein, x is a set {1, 2}, T 1x represents two pose solutions T 11 and T 12 of the left eye, and T 2x represents two pose solutions T 21 and T 22 of the right eye, represents the relative pose parameters between the left and right eyes.

[0070] Step S303; since the pose between the binoculars is known, the obtained T is compared with the known relative pose parameters between the binoculars, so as to determine the correct pose solution of the left and right eyes respectively; for example, T 11 is the correct pose solution of the left eye, and T 22 is the correct pose solution of the right eye.

[0071] Step S304; according to the calculation result of T22*(T11)-1 and the known relative pose parameters between the binoculars, the error is determined, and whether the pose solving result of the left and right eyes relative to the planar marker board meets the preset accuracy requirement is determined according to the error. If it meets the preset accuracy requirement, step S305 is performed, otherwise, step S306 is performed.

[0072] Step S305; when the pose solving result of the left and right eyes relative to the planar marker board meets the above-mentioned preset accuracy requirement, the pose solving result of the left and right eyes relative to the planar marker board is taken as the pose solving result of the current frame.

[0073] Step S306; when the pose solving result of the left and right eyes relative to the planar marker board does not meet the above-mentioned preset accuracy requirement, a region of interest of the planar marker board is selected, and a brightness equalization process is performed on the region of interest.

[0074] The determination of whether the pose solving result of the binoculars relative to the planar marker board meets the preset accuracy requirement is continuously performed, and when the pose solving result meets the above-mentioned preset accuracy requirement, step S305 is performed, otherwise, step S307 is performed.

[0075] Step S307; when the pose solving result of the binoculars relative to the planar marker board does not meet the above-mentioned preset accuracy requirement, a photosensitive sensor is used to detect the current light condition; when the light is overexposed or insufficient, the corresponding adjustment scheme (for example, adjusting the exposure time and / or gain of the visual sensor, adjusting the light supplement intensity using a light supplement device, etc.) is continuously performed, which is specifically described in steps S205 to S210 in Figure 2 , and will not be described here. After the planar marker board is detected by the left and right eyes, step S301 is returned to be performed, and the determination of whether the pose solving result of the left and right eyes meets the preset accuracy requirement is continuously performed.

[0076] According to the embodiment of the present application, a pose solving processing device based on a multi-eye visual sensor is further provided.

[0077] Figure 4 is a structural block diagram of the pose solving processing device based on the multi-eye visual sensor according to the embodiment of the present application. As shown in Figure 4 , the pose solving processing device based on the multi-eye visual sensor comprises: a detection module 40, configured to detect a marker using a multi-eye visual sensor in a current frame; an adjustment module 42, configured to trigger an adjustment processing procedure of a current environment to be performed when only no more than one visual sensor in the multi-eye visual sensor detects the marker; a judgment module 44, configured to judge whether a pose solving result of at least two eyes relative to the marker meets a preset accuracy requirement according to a relative pose parameter between the at least two eyes when the at least two eyes in the multi-eye visual sensor both detect the marker; and a determination module 46, configured to take the pose solving result of the at least two eyes relative to the marker as the pose solving result of the current frame when the pose solving result of the at least two eyes relative to the marker meets the above-mentioned preset accuracy requirement.

[0078] is adopted Figure 4The multi-vision sensor-based pose solving processing apparatus shown, when the detection module 40 detects the marker using the multi-vision sensor, the adjustment module 42 triggers the execution of the adjustment processing procedure of the current environment in the case that no more than one vision sensor (only 1 vision sensor or 0 vision sensor) of the multi-vision sensor detects the marker. By setting the adjustment processing procedure, the problem that the marker is detected based on the camera in the related art and is easily affected by the environmental conditions, resulting in a reduced recognition rate, can be solved. Moreover, when at least two (2 or more vision sensors) of the multi-vision sensor detect the marker, the judgment module 44 can determine whether the pose solving result of the at least two relative to the marker meets the preset accuracy requirement according to the relative pose parameters between the at least two. When the pose solving result of the at least two relative to the marker meets the preset accuracy requirement, the determination module 46 takes the pose solving result of the at least two relative to the marker as the pose solving result of the current frame. Through the accuracy judgment processing scheme, the pose solving accuracy is improved.

[0079] Preferably, the adjustment module 42 can further include a detection unit (not shown in the specification) for detecting the current light condition using a photosensitive sensor, a first execution unit (not shown in the specification) for adjusting the exposure time and / or gain of the vision sensor that does not detect the marker when the light is overexposed, and then continuing to detect the marker, and cyclically executing the step until at least two of the multi-vision sensor detect the marker, a second execution unit (not shown in the specification) for adjusting the light supplement intensity using a light supplement device when the light is insufficient, and then continuing to detect the marker, and cyclically executing the step until at least two of the multi-vision sensor detect the marker. Figure 3 Figure 3 Figure 3

[0080] It should be noted that the preferred implementation of the combination of each module and each unit in the multi-vision sensor-based pose solving processing apparatus can be specifically referred to the description of Figures 1 to 3 , which will not be described here again.

[0081] According to the embodiment of the present application, a multi-vision sensor-based pose solving processing device is further provided.

[0082] Figure 5 is a structural block diagram of the multi-vision sensor-based pose solving processing device according to the preferred embodiment of the present application. As Figure 5 ​​​As shown, the multi-vision sensor-based pose solving processing device includes a multi-vision sensor 50, a light sensor 52, a light supplement device 54, a memory 56, and a processor 58. The multi-vision sensor 50 is configured to detect a marker. The light sensor 52 is configured to detect a current light condition. The light supplement device 54 is configured to supplement light when the light is insufficient. The memory 56 is configured to store computer execution instructions. The processor 58 is configured to execute the computer execution instructions stored in the memory, so that the robot executes the pose solving processing method described above. For details, refer to the description of Figures 1 to 3 the same implementation and principles, which will not be described herein.

[0083] In summary, for the problem of the camera-based marker detection technology in the related art being easily affected by environmental conditions (especially in adverse environmental light (e.g., darkness, dimness, strong light, etc.)), having a low recognition rate, and the pose solving accuracy of the robot relative to the marker being insufficient, the above embodiments provided by the present application use a multi-vision (e.g., binocular, trinocular, etc.) sensor to detect a marker. When only one or more than one vision sensor detects the marker, an adjustment processing procedure for the current environment is triggered to be executed. By setting the adjustment processing procedure, the problem of the low recognition rate can be solved. Moreover, when at least two (2 or more) of the multi-vision sensor detect the marker, whether the pose solving result of the at least two relative to the marker meets a preset accuracy requirement can be determined according to the relative pose parameters between the at least two. When the pose solving result of the at least two relative to the marker meets the preset accuracy requirement, the pose solving result of the at least two relative to the marker is taken as the pose solving result of the current frame. Through the above accuracy determination processing scheme, the pose solving accuracy is improved. Moreover, for the two-unknown-solution problem of the vision sensor pose solving, the advantage of the multi-vision is also utilized. The relative pose parameters between the at least two are used to find the correct pose solution, further improving the solving accuracy. In addition, a multi-frame fusion method is also used. The mean value of the multi-frame pose solving results is taken as the final pose solving result, further improving the accuracy and stability.

[0084] The above disclosure is only some specific embodiments of the present application, but the present application is not limited thereto. Any changes that can be thought of by those skilled in the art shall fall within the protection scope of the present application.

Claims

1. A pose solving method based on a multi-view vision sensor, characterized in that, Comprising: detecting a marker using a multi-view vision sensor in a current frame; in a case that no more than one vision sensor in the multi-view vision sensor detects the marker, triggering a current environment adjustment processing procedure, wherein the current environment adjustment processing procedure comprises: detecting a current light condition using a light sensor, when light is overexposed, adjusting an exposure time and / or a gain of the vision sensor, and then continuing to detect the marker, and the step is repeatedly executed until at least two views in the multi-view vision sensor detect the marker, when light is insufficient, adjusting an intensity of a light supplement device using the light supplement device, and then continuing to detect the marker, and the step is repeatedly executed until at least two views in the multi-view vision sensor detect the marker, and before detecting the current light condition using the light sensor, further comprising: performing a histogram equalization processing on the marker, continuing to detect the marker, and the step is repeatedly executed, and judging whether at least two views in the multi-view vision sensor detect the marker; when at least two views in the multi-view vision sensor detect the marker, judging whether a pose solving result of the at least two views relative to the marker satisfies a preset accuracy requirement according to a relative pose parameter between the at least two views; when the pose solving result of the at least two views relative to the marker satisfies the preset accuracy requirement, taking the pose solving result of the at least two views relative to the marker as a pose solving result of a current frame.

2. The method of claim 1, wherein, After judging whether the pose solving result of the at least two views relative to the marker satisfies the preset accuracy requirement, further comprising: when the pose solving result of the at least two views relative to the marker does not satisfy the preset accuracy requirement, selecting a region of interest of the marker, and performing a brightness equalization processing on the region of interest; continuing to judge whether the pose solving result of the at least two views relative to the marker satisfies the preset accuracy requirement, and when the pose solving result of the at least two views relative to the marker satisfies the preset accuracy requirement, taking the pose solving result of the at least two views relative to the marker as the pose solving result of the current frame.

3. The method of claim 2, wherein, After continuing to judge whether the pose solving result of the at least two views relative to the marker satisfies the preset accuracy requirement, further comprising: when the pose solving result of the at least two views relative to the marker does not satisfy the preset accuracy requirement, detecting a current light condition using a light sensor; when light is overexposed, adjusting an exposure time and / or a gain of the vision sensor, and then continuing to detect the marker, and the step is repeatedly executed until at least two views in the multi-view vision sensor detect the marker; when light is insufficient, adjusting an intensity of a light supplement device using the light supplement device, and then continuing to detect the marker, and the step is repeatedly executed until at least two views in the multi-view vision sensor detect the marker.

4. The method according to claim 1 or 3, characterized in that, Further comprising: When the light is overexposed, and it is determined that the at least two eyes of the multi-view visual sensor cannot detect the marker by adjusting the exposure time and / or gain of the visual sensor, the histogram of the marker is equalized, and the detection of the marker is continued, and the step is repeatedly executed until the at least two eyes of the multi-view visual sensor detect the marker.

5. The method according to claim 1 or 3, characterized in that, Further comprising: When the light is insufficient, and it is determined that the at least two eyes of the multi-view visual sensor cannot detect the marker by adjusting the light supplement intensity using the light supplement device, the exposure time and / or gain of the visual sensor is adjusted, and then the detection of the marker is continued, and the step is repeatedly executed to determine whether the visual sensor that does not detect the marker detects the marker; When it is determined that the at least two eyes of the multi-view visual sensor cannot detect the marker by adjusting the exposure time and / or gain of the visual sensor, the histogram of the marker is equalized, and then the detection of the marker is continued, and the step is repeatedly executed until the at least two eyes of the multi-view visual sensor detect the marker.

6. The method of claim 1, wherein, Determining whether the pose solution result of the at least two eyes relative to the marker meets the preset accuracy requirement according to the relative pose parameters between the at least two eyes comprises: For each visual sensor in the at least two eyes, a pose solution result of the visual sensor relative to the marker is obtained, and each visual sensor corresponds to two pose solutions; According to the relative pose parameters between the two visual sensors in the at least two eyes, and the pose solutions corresponding to the two visual sensors, it is determined whether the pose solution result of the two visual sensors relative to the marker meets the preset accuracy requirement.

7. The method of claim 6, wherein, According to the relative pose parameters between the two visual sensors, and the pose solutions corresponding to the two visual sensors, it is determined whether the pose solution result of the two visual sensors relative to the marker meets the preset accuracy requirement comprises: The relative pose parameters between the two visual sensors are calculated by the following formula: Wherein, x is a set {1, 2}, T1x represents two pose solutions corresponding to the first visual sensor of the two visual sensors, T2x represents two pose solutions corresponding to the second visual sensor of the two visual sensors, and the The relative pose parameters between the two visual sensors are represented by T. The calculated acquisition The relative pose parameters between the two visual sensors are compared with the predetermined two visual sensors, and the correct pose solutions corresponding to the two visual sensors are determined respectively. According to the correct pose solution The error between the relative pose parameters of the two vision sensors and the predetermined values ​​is used to determine whether the pose calculation results of the two vision sensors meet the preset accuracy requirements.

8. The method of claim 1, wherein, Further comprising: The pose solution processing method of claim 1 is repeatedly executed until the pose solution result of the at least two eyes relative to the marker in multiple frames is obtained; For each of the at least two frames closest to the current frame in the multiple frames, the pose solution result of the frame is converted to the same coordinate system, and the mean value of the converted pose solution result is obtained; According to the mean value of the pose solution result of each frame in the at least two frames, a first mean value is obtained, and the first mean value is taken as the final pose solution result.

9. The method of claim 1, wherein, Further comprising: In the case that only one visual sensor in the multi-view visual sensor detects the marker, a monocular pose solution result of the visual sensor relative to the marker is saved; When it is determined that the at least two eyes of the multi-view visual sensor cannot detect the marker by executing the adjustment processing procedure of the current environment, the saved monocular pose solution result is taken as the pose solution result of the current frame; After performing multi-frame pose solving processing, when the pose solving result of the at least two eyes relative to the marker does not appear in each saved frame pose solving result, a second mean value is calculated for monocular pose solving results of the at least two frames closest to the current frame, and the second mean value is taken as the final pose solving result.

10. A multi-view vision sensor based pose solving processing apparatus, characterized by, Comprise: The detection module is used for detecting the marker by using the multi-eye vision sensor in the current frame; The adjustment module is used for triggering the execution of the adjustment processing procedure of the current environment when no more than one vision sensor in the multi-eye vision sensor detects the marker, wherein the adjustment module comprises: a detection unit used for detecting the current light condition by using a photosensitive sensor; a first execution unit used for adjusting the exposure time and / or gain of the vision sensor when the light is overexposed, and then continuing to detect the marker, and the step is repeatedly executed until at least two eyes in the multi-eye vision sensor detect the marker; a second execution unit used for adjusting the light supplement intensity by using a light supplement device when the light is insufficient, and then continuing to detect the marker, and the step is repeatedly executed until at least two eyes in the multi-eye vision sensor detect the marker, and before the detection unit detects the current light condition by using the photosensitive sensor, the adjustment module is further used for: performing histogram equalization processing on the marker, continuing to detect the marker, and repeatedly executing the step to determine whether at least two eyes in the multi-eye vision sensor detect the marker; The judgment module is used for judging whether the pose solving result of the at least two eyes relative to the marker meets a preset accuracy requirement according to the relative pose parameters between the at least two eyes when the at least two eyes in the multi-eye vision sensor detect the marker; The determination module is used for taking the pose solving result of the at least two eyes relative to the marker as the pose solving result of the current frame when the pose solving result of the at least two eyes relative to the marker meets the preset accuracy requirement.

11. A multi-view vision sensor based pose solving processing device, comprising: Multi-eye vision sensor, photosensitive sensor, light supplement device, memory and processor, characterized in that, The multi-eye vision sensor is used for detecting the marker; The photosensitive sensor is used for detecting the current light condition; The light supplement device is used for supplementing light when the light is insufficient; The memory is used for storing computer execution instructions; The processor is used for executing the computer execution instructions stored in the memory, so that the robot executes the method in any one of claims 1 to 9.

Images