A pose data acquisition method, device, equipment and medium

Abstract

The application discloses a pose data acquisition method and device, equipment and medium, and relates to the field of computer vision and robots. The method comprises the following steps: determining a target sampling viewpoint on a spherical surface with the center of a target object as the spherical center and a preset sampling distance as the radius; determining a target motion trajectory of a mechanical arm carrying a camera at the end based on the target sampling viewpoint and by using a preset trajectory planning method; controlling the mechanical arm to move in the target motion trajectory, and recording a video of the target object by using the camera carried at the end of the mechanical arm to obtain a target video; extracting a target image frame from the target video, and determining pose information corresponding to the target image frame by using a preset visual SLAM algorithm. The acquired pose data has the characteristic of uniform distribution, so that the pose information of the target object extracted finally is more complete, and great help can be provided for the development, test, verification and improvement of a 6D pose estimation algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and robotics, in particular to a pose data acquisition method, device, equipment and medium. BACKGROUND

[0002] The prior art method for acquiring 6D pose data of an object needs to place a plurality of two-dimensional codes around the object to obtain the pose data of the object. Since the relative pose of the object with respect to each two-dimensional code is fixed during image acquisition, only the two-dimensional codes with independent IDs (i.e., Identification, unique identification) need to be identified in different acquired images to obtain a data set with different 6D poses of the object. Disadvantages: the data obtained by this acquisition method has the background of the object covered by the two-dimensional codes, which cannot be changed, and there is still a certain distance from the real scene background in which the object is placed.

[0003] The existing Elasticfusion method needs to acquire a continuous video with the object to be acquired, and then a dense scene three-dimensional point cloud map is built by the SLAM (i.e., Simultaneous Localization and Mapping, simultaneous localization and mapping) mapping method of Elasticfusion. Finally, the three-dimensional model of the object to be acquired is aligned with the object point cloud in the three-dimensional point cloud map by manual annotation, so that the continuous video can be converted into complete 6D pose data of the object. Since the object can be placed in any background, the data obtained by this method is closer to the real scene. Disadvantages: it cannot work effectively outdoors; Elasticfusion cannot work effectively in a scene with a large indoor depth; Elasticfusion cannot work effectively in a scene with strong light absorption and strong light reflection, because the RGBD camera cannot obtain effective depth and point cloud in such a situation. And generally handheld around shooting, the object poses in a single video are relatively arbitrary, single, and the pose data is not uniformly covered.

[0004] From the above, in the process of acquiring 6D pose data, how to avoid the situation that the acquired pose data is not accurate and not uniformly distributed is a problem to be solved in the field. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a pose data acquisition method, device, equipment and medium, which can make the acquired pose data have the characteristic of uniform distribution, so that the pose information of the target object extracted finally is more complete, which can provide great help for the development, test, verification and improvement of 6D pose estimation algorithm. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a pose data acquisition method, comprising:

[0007] determining a target sampling viewpoint on a sphere with a center of the target object as a sphere center and a preset sampling distance as a radius;

[0008] determining a target motion trajectory of a robot arm carrying a camera at an end thereof based on the target sampling viewpoint and by using a preset trajectory planning method;

[0009] controlling the robot arm to move according to the target motion trajectory and recording a video of the target object by using the camera carried at the end of the robot arm, to obtain a target video; wherein a ray emitted forward by the camera, which is perpendicular to a pixel plane of the camera and passes through an optical center, always passes through the center of the target object;

[0010] extracting a target image frame from the target video and determining pose information corresponding to the target image frame by using a preset visual SLAM algorithm.

[0011] Optionally, the determining of the target sampling viewpoint on the sphere with the center of the target object as the sphere center and the preset sampling distance as the radius comprises:

[0012] determining the target sampling viewpoint on the sphere with the center of the target object as the sphere center and the preset sampling distance as the radius based on a preset horizontal sampling density value and a preset vertical sampling density value.

[0013] Optionally, the determining of the target motion trajectory of the robot arm carrying the camera at the end thereof based on the target sampling viewpoint and by using the preset trajectory planning method comprises:

[0014] determining the target motion trajectory of the robot arm carrying the camera at the end thereof based on the target sampling viewpoint and by using a fast search random number method.

[0015] Optionally, the extracting of the target image frame from the target video and the determining of the pose information corresponding to the target image frame by using the preset visual SLAM algorithm comprises:

[0016] clipping a video segment for adjusting a pose of the robot arm from the target video by using a preset video automatic clipping script, to generate a clipped video;

[0017] extracting a target image frame from the clipped video and determining pose information corresponding to the target image frame by using a preset visual SLAM algorithm.

[0018] Optionally, the clipping of the video segment for adjusting the pose of the robot arm from the target video by using the preset video automatic clipping script comprises:

[0019] determine a video frame key point for adjusting the pose of the mechanical arm by using a preset visual detection algorithm, and determine a video clip to be cut based on the video frame key point;

[0020] Or, a setting instruction for setting the video frame key point for adjusting the pose of the mechanical arm in the target video is received, and a video clip to be cut is determined based on the setting instruction.

[0021] Optionally, before determining the target motion trajectory of the mechanical arm with the camera at the end thereof by using a preset trajectory planning method based on the target sampling viewpoint, the method further comprises:

[0022] obtaining a coordinate conversion relationship between the mechanical arm and the camera carried at the end of the mechanical arm by using a hand-eye calibration method, or determining the coordinate conversion relationship between the mechanical arm and the camera carried at the end of the mechanical arm by using a SolidWorks macro tool;

[0023] determining a target setting parameter for setting the mechanical arm based on the coordinate conversion relationship, and setting the target setting parameter for the mechanical arm.

[0024] Optionally, the method further comprises:

[0025] determining the target motion trajectory of the mechanical arm with the camera at the end thereof and a running speed of the mechanical arm by using a preset trajectory planning method;

[0026] Correspondingly, the method further comprises:

[0027] controlling the camera to start video recording, and controlling the mechanical arm to move according to the target motion trajectory at the running speed;

[0028] controlling the mechanical arm to pause movement when the mechanical arm moves to a target sampling viewpoint, controlling a rotating device for rotating the camera on the mechanical arm to rotate the camera by 360 degrees, and controlling the mechanical arm to continue moving according to the target motion trajectory at the running speed when the rotating device rotates back to an initial position, jumping to the step of controlling the mechanical arm to pause movement when the mechanical arm moves to a next target sampling viewpoint, and ending until the mechanical arm moves according to the target motion trajectory.

[0029] controlling the camera to stop recording to obtain the target video recorded by the camera.

[0030] In a second aspect, the present application discloses a pose data acquisition device, comprising:

[0031] a sampling viewpoint determination unit, configured to determine a target sampling viewpoint from a sphere with a center of the target object as a sphere center and a preset sampling distance as a radius;

[0032] a motion trajectory determination unit, configured to determine a target motion trajectory of a robot arm carrying a camera at an end thereof based on the target sampling viewpoint and by using a preset trajectory planning method;

[0033] a target video recording unit, configured to control the robot arm to move according to the target motion trajectory, and record a target video of the target object by using the camera carried at the end of the robot arm, so as to obtain the target video; wherein a ray emitted by the camera in a forward direction, which is perpendicular to a pixel plane of the camera and passes through an optical center, always passes through the center of the target object;

[0034] a pose data determination unit, configured to extract a target image frame from the target video, and determine pose information corresponding to the target image frame by using a preset visual SLAM algorithm.

[0035] In a third aspect, the present application discloses an electronic device, comprising:

[0036] a memory, configured to save a computer program;

[0037] a processor, configured to execute the computer program to implement the aforementioned pose data acquisition method.

[0038] In a fourth aspect, the present application discloses a computer storage medium, configured to save a computer program; wherein the computer program is executed by a processor to implement the steps of the aforementioned pose data acquisition method.

[0039] In the present application, first determine the target sampling viewpoint on the spherical surface with the center of the target object as the spherical center and the preset sampling distance as the radius; determine the target motion trajectory of the mechanical arm carrying the camera at the end based on the target sampling viewpoint and using a preset trajectory planning method; control the mechanical arm to move in the target motion trajectory, and use the camera carried at the end of the mechanical arm to record a video of the target object to obtain a target video; wherein the ray emitted by the camera in front and perpendicular to the camera pixel plane and passing through the optical center always passes through the center of the target object; extract a target image frame from the target video, and determine the corresponding pose information of the target image frame using a preset visual SLAM algorithm. In this way, the present application uses the independently designed "spherical surface sampling" method to make the mechanical arm drive the camera at the end to move according to the specifically planned spherical surface trajectory, and ensures that the camera is always aligned with the object to be collected during the movement, and finally extracts the pose data of the target object from the target video shot by the camera. Since the pose data collected by the mechanical arm spherical surface sampling method in the present application has the characteristic of uniform distribution, the pose information of the target object extracted finally is more complete, which can provide great help for the development, test, verification and improvement of 6D pose estimation algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0041] Figure 1 A pose data acquisition method flowchart is provided for the present application;

[0042] Figure 2 A sampling viewpoint schematic diagram is provided for the present application;

[0043] Figure 3 A coordinate pointing schematic diagram is provided for the present application;

[0044] Figure 4 A three-dimensional model graph and a point cloud graph comparison graph are provided for the present application;

[0045] Figure 5 A specific pose data acquisition method flowchart is provided for the present application;

[0046] Figure 6 A pose data acquisition device structure schematic diagram is provided for the present application;

[0047] Figure 7An electronic device structure diagram is provided in the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0049] In the prior art, the 6D pose data of an object can be collected by using a two-dimensional code, but the data obtained by this collection method is covered by the background of the object, cannot be changed, and still has a certain distance from the scene background in which the real object is placed. The existing Elasticfusion method cannot work effectively outdoors; ElasticFusion cannot work effectively in a scene with a large indoor depth; ElasticFusion cannot work effectively in a scene with strong light absorption and strong light reflection, and generally, a handheld camera is used to take a video, and the pose of the object in the video is relatively random and single. The pose data is not uniformly covered, and uniform samples cannot be obtained for subsequent training. In the present application, the collected pose data has the characteristic of uniform distribution, so that the pose information of the target object extracted finally is more complete, which can provide great help for the development, testing, verification and improvement of the 6D pose estimation algorithm.

[0050] The embodiment of the present application discloses a pose data collection method, referring to Figure 1 The method comprises the following steps:

[0051] Step S11: determining a target sampling viewpoint on a spherical surface determined by taking the center of the target object as the spherical center and taking a preset sampling distance as the radius.

[0052] In the present application, the user first needs to build a scene similar to the final real application scene on a horizontal desktop with a certain height to provide a real background for the data. Then, a single or several target objects to be collected are placed in this background.

[0053] It can be understood that the spherical surface in the embodiment is determined by the center of the target object. In a specific embodiment, when the target object is a single object, the centroid of the target object can be taken as the spherical center of the spherical surface; when the target object is a plurality of objects, the centers of the plurality of objects can be taken as the spherical center of the spherical surface. That is, we generate each sampling viewpoint on the spherical surface by surrounding the centroid of a single object or the centers of a plurality of objects. For example, Figure 2As shown in the sampling viewpoint schematic diagram provided by the application, each sampling viewpoint is located on a spherical surface with the center of the target object as the spherical center and with the preset sampling distance as the radius, and the horizontal axis and the vertical axis of the three-dimensional rectangular coordinate corresponding to each sampling viewpoint are tangent to the spherical surface, and the positive direction of the vertical axis points to the spherical center.

[0054] It should be noted that the preset sampling distance can be changed according to different sampling objects or scenes, and when changed, it is necessary to ensure that the spherical surface with the preset sampling distance as the radius is within the movable range of the mechanical arm. In specific implementation, the preset sampling distance is generally set to 0.3m to 1m.

[0055] In the embodiment, the target sampling viewpoint determined from the spherical surface with the center of the target object as the spherical center and with the preset sampling distance as the radius can include: determining the target sampling viewpoint from the spherical surface with the center of the target object as the spherical center and with the preset sampling distance as the radius based on a preset horizontal sampling density value and a preset vertical sampling density value.

[0056] In the embodiment, the preset horizontal sampling density value and the preset vertical sampling density value can also be understood as the latitude sampling density value and the longitude sampling density value, which are used to determine the relative position relationship between the sampling viewpoints on the spherical surface. In a specific implementation, the preset horizontal sampling density value can be set to 15 degrees, and the longitude sampling density value can be set to 15 degrees, so that the latitude difference between each horizontal sampling annulus on the spherical surface is 15 degrees, and the longitude difference between each vertical sampling annulus on the spherical surface is 15 degrees. In the embodiment, when the target sampling viewpoint is determined, the number of target sampling viewpoints on each annulus can be determined based on the single-annulus sampling number defined by the user, and in a preferred implementation, the number of target sampling viewpoints on each annulus is consistent, and preferably 24. The number can be changed according to different sampling scenes.

[0057] Step S12: determining the target motion trajectory of the mechanical arm with the camera at the end based on the target sampling viewpoint and using a preset trajectory planning method.

[0058] In the embodiment, the end of the mechanical arm is provided with a camera, the camera is preferably an RGBD camera, and the fixing method of the camera is preferably to fix a flange plate printed based on 3D printing technology at the end of the mechanical arm. The mechanical arm is preferably a UR5 mechanical arm.

[0059] It should be noted that before planning the motion trajectory of the mechanical arm, the parameters of the mechanical arm are first set based on the relative position relationship between the mechanical arm and the camera. When the mechanical arm moves with the camera for shooting in the subsequent process, it must be ensured that the optical axis of the camera always passes through the center of mass of the object, so as to ensure that the object is always in the sampling field of view, and at the same time, the background information of the object can also be shot. In addition, in the trajectory planning of the embodiment, the camera on the mechanical arm should be allowed to shoot the object from various angles as much as possible. In addition, the residence time at each angle should also be constant, so as to ensure that the number of samples at each angle in the final collected pose data is consistent.

[0060] In the specific embodiment, the preset trajectory planning method can be a rapid-exploration random tree (RRT) method. The target motion trajectory includes joint motion angles corresponding to each joint of the mechanical arm in continuous time. In the preferred embodiment, the target motion trajectory can be generated in a ring-shaped manner to traverse the target sampling viewpoint, so as to ensure that the mechanical arm passes through each target sampling viewpoint based on the ring-shaped route. In the specific embodiment, the mechanical arm can be set to traverse each target sampling viewpoint in a transverse ring-shaped manner, that is, when the first ring-shaped motion is completed, the traversal continues on the ring-shaped route at the next latitude, so as to finally ensure the uniformity and comprehensiveness of the sampling.

[0061] Step S13: controlling the mechanical arm to move in the target motion trajectory, and using the camera carried at the end of the mechanical arm to record a video of the target object to obtain a target video; wherein a ray emitted by the camera in a forward direction, which is perpendicular to the pixel plane of the camera and passes through the optical center, always passes through the center of the target object.

[0062] In this embodiment, the method for determining the target motion trajectory of the mechanical arm carrying the camera at the end can comprise: determining the target motion trajectory of the mechanical arm carrying the camera at the end and the running speed of the mechanical arm by using the preset trajectory planning method; correspondingly, the method for controlling the mechanical arm to move along the target motion trajectory and recording a video of the target object by using the camera carried at the end of the mechanical arm to obtain a target video can comprise: controlling the camera to start video recording and controlling the mechanical arm to move along the target motion trajectory at the running speed; when the mechanical arm moves to a target sampling viewpoint, controlling the mechanical arm to pause movement, controlling a rotating device on the mechanical arm for rotating the camera to rotate the camera by 360 degrees, and when the rotating device rotates back to the initial position, controlling the mechanical arm to continue moving along the target motion trajectory at the running speed; when the mechanical arm moves to the next target sampling viewpoint, jumping to the step of controlling the mechanical arm to pause movement, until the mechanical arm moves along the target motion trajectory ends; and controlling the camera to stop recording to obtain the target video recorded by the camera.

[0063] The camera on the mechanical arm can be rotated by using the rotating device. In a specific embodiment, a rotatable flange can be selected as the rotating device.

[0064] In a preferred embodiment, when the target motion trajectory is generated, the motion speed of the mechanical arm can be determined simultaneously, and when the mechanical arm reaches each target sampling viewpoint, the rotating device on the mechanical arm for rotating the camera can be controlled to rotate the camera by 360 degrees to capture images of the target object at different angles of the current target sampling viewpoint, thereby ensuring the comprehensiveness of sampling. It can be understood that the initial position can be defined on the rotating device in advance, and each time the rotating device rotates back to the initial position, the mechanical arm is controlled to move in the direction of the next target sampling viewpoint according to the target motion trajectory until the target motion trajectory is completed. Figure 3 FIG. 1 shows a coordinate pointing diagram when the camera rotates and samples, in which, under the camera view, at each sampling viewpoint, the horizontal axis and the vertical axis corresponding to each sampling time point of each viewpoint are determined to be tangent to the sphere, and the positive direction of the vertical axis points to the center of the sphere, so that each sampling viewpoint generates a snowflake-like coordinate pointing corresponding to the three-dimensional rectangular coordinate. Figure 3

[0065] Step S14: extracting a target image frame from the target video and determining pose information corresponding to the target image frame by using a preset visual SLAM algorithm.

[0066] ​In the embodiment, the preset visual SLAM algorithm can be used to extract image frames and pose information corresponding to each view angle of the spherical surface to determine the pose information corresponding to the target image frame.

[0067] In the specific embodiment of the present application, after determining the pose information corresponding to the target image frame, a dataset can be created based on the Labelfusion method using the SLAM mapping-artificial labeling method based on Elasticfusion, so that the 6D pose dataset of the object with color, depth, mask diagram and 6D pose data can be obtained. Since the pose data collected by the mechanical arm spherical sampling method in the present application has the characteristic of uniform distribution, the finally generated 6D pose dataset has the characteristics of completeness and reduces randomness, which is close to the real scene of the final application. In addition, the present application is easy to use, accurate and fast, which can provide support for the automatic creation method and can provide great help for the development, testing, verification and improvement of the 6D pose estimation algorithm.

[0068] In the embodiment, after determining the pose information corresponding to the target image frame, the target video can be used to perform three-dimensional mapping using the Elasticfusion algorithm, and a three-dimensional point cloud consistent with the real scene can be reconstructed. In the specific embodiment, the point cloud diagram can be created by using the ICP algorithm (i.e. Iterative Closest Point, Iterative Closest Point) to obtain the pose transformation relationship of the point corresponding to the point position of the target object in the first video frame in the target video in the video by manually labeling the point position using the Labelfusion program. For example, Figure 4 The right image is a three-dimensional model diagram of a white paper box, and the left image is a point cloud diagram reconstructed based on the real scene.

[0069] In this embodiment, first, a target sampling viewpoint is determined on a sphere with the center of the target object as the sphere center and a preset sampling distance as the radius; a target motion trajectory of a mechanical arm carrying a camera at the end is determined based on the target sampling viewpoint and by using a preset trajectory planning method; the mechanical arm is controlled to move in the target motion trajectory, and the target object is video-recorded by using the camera carried at the end of the mechanical arm to obtain a target video; wherein a ray emitted forward by the camera, which is perpendicular to the camera pixel plane and passes through the optical center, always passes through the center of the target object; target image frames are extracted from the target video, and corresponding pose information of the target image frames is determined by using a preset visual SLAM algorithm. In this way, in this embodiment, the target motion trajectory of the mechanical arm carrying the camera at the end is determined after a sphere is generated based on the target object and a target sampling viewpoint is determined on the sphere, the mechanical arm carrying the camera is used to sequentially pass through each target sampling viewpoint on the sphere, and the pose information of the target object is extracted from the target video recorded by the camera. Since the pose data collected by the mechanical arm sphere sampling method in this application has the characteristic of uniform distribution, the pose information of the target object extracted finally is more complete, which can provide great help for the development, test, verification and improvement of 6D pose estimation algorithm. In addition, by using the target image frames and the corresponding pose information, an article 6D pose dataset with color, depth, mask diagram and 6D pose data can also be obtained, and the finally generated 6D pose dataset has completeness, reduces randomness, is close to the real scene of the final application, and has strong usability.

[0070] Figure 5 A specific pose data collection method flowchart is provided for the embodiments of the application. Referring to FIG. 21, the method comprises the following steps. Figure 5

[0071] Step S21: determining a target sampling viewpoint on a sphere with the center of the target object as the sphere center and a preset sampling distance as the radius.

[0072] Wherein, the more specific processing process of step S21 can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0073] Step S22: obtaining the coordinate conversion relationship between the mechanical arm and the camera carried at the end of the mechanical arm by using the method of hand-eye calibration; or, determining the coordinate conversion relationship between the mechanical arm and the camera carried at the end of the mechanical arm by using the method of SolidWorks macro tool; determining a target setting parameter for setting the mechanical arm based on the coordinate conversion relationship, and setting the target setting parameter for the mechanical arm.

[0074] ​It can be understood that, in the embodiment, since the camera is mounted at the end of the mechanical arm, when planning the motion trajectory of the mechanical arm, the relative positional relationship between the mechanical arm and the camera is considered, and the coordinate conversion relationship between the mechanical arm and the camera carried at the end of the mechanical arm can be obtained by using the hand-eye calibration method; meanwhile, the coordinate conversion relationship (external parameter) between the mechanical arm and the camera carried at the end of the mechanical arm can also be determined by using the SolidWorks macro tool, so that the TCP parameter of the mechanical arm is set, and the background information of the object can be shot while ensuring that the object is always in the sampling field of view.

[0075] Step S23: determining the target motion trajectory of the mechanical arm carrying the camera at the end based on the target sampling viewpoint and by using the fast search random number method.

[0076] It can be understood that, in the embodiment, the target motion trajectory of the mechanical arm carrying the camera at the end can be determined by using the fast search random number method. In the specific embodiment, the mechanical arm may, due to the mechanical design limitation of itself, not be able to continuously move through all points on the spherical surface at one time, and a large range of mechanical arm posture adjustment is needed when moving between adjacent viewing angles on the spherical surface, which can also be understood as that the joint angle of the mechanical arm will change greatly, and this will directly lead to that the Elasticfusion angle point changes too fast or is missing, so that the point cloud graph may not be created based on the finally determined pose data.

[0077] Therefore, the fast search random number method is used for motion planning of the mechanical arm in the embodiment. Since the RRT method cannot obtain an optimal solution, a plurality of results returned by using the RRT are sorted in ascending order according to the number of trajectory points contained in the results, and a group of solutions with the least number is selected as the optimal trajectory for movement of two adjacent viewpoints on the spherical surface, so as to ensure that the posture of the camera on the mechanical arm changes as a whole, thereby better ensuring the effect of scene reconstruction. When the point cloud graph needs to be constructed subsequently, the continuity of the inter-frame images in the Elasticfusion mapping process is ensured, and a certain similarity exists between the two consecutive images, so that the camera tracking and subsequent point cloud reconstruction are smoothly completed, and the situation that the reconstructed point cloud graph has a large error and is quite different from the real scene is avoided, and the accuracy of the final 6D data set is ensured when the 6D data set needs to be created subsequently.

[0078] Step S24: controlling the mechanical arm to move in the target motion trajectory, and recording a video of the target object by using the camera carried at the end of the mechanical arm, to obtain a target video; wherein the ray emitted forward by the camera, which is perpendicular to the pixel plane of the camera and passes through the optical center, always passes through the center of the target object.

[0079] The more specific processing procedure about step S24 can refer to the corresponding content disclosed in the foregoing embodiments, and will not be described here in detail.

[0080] Step S25: cutting the video segment for adjusting the pose of the mechanical arm in the target video by using the preset video automatic editing script to generate a cut video.

[0081] In this embodiment, the cutting of the video segment for adjusting the pose of the mechanical arm in the target video by using the preset video automatic editing script can include: determining the video frame key points for adjusting the pose of the mechanical arm by using a preset visual detection algorithm, and determining the video segment to be cut based on the video frame key points; or, receiving a setting instruction for setting the video frame key points for adjusting the pose of the mechanical arm in the target video, and determining the video segment to be cut based on the setting instruction.

[0082] In this embodiment, when transforming between trajectories at different latitudes in the automatic spherical sampling process, the mechanical arm will make large-scale pose adjustment due to its own design characteristics in order to make the subsequent trajectory reachable. In this embodiment, by cutting the recorded continuous video, all segments of large-scale adjustment of the pose of the mechanical arm are cut, and a complete continuous spherical sampling video is obtained.

[0083] Specifically, when performing video editing, a full-automatic editing method or a semi-automatic editing method of manual annotation editing can be used to improve the proportion of effective image frames in the target video, to ensure that the image frames and the pose information of the image frames generated finally can successfully create a three-dimensional point cloud map, so that the scene reconstruction is better realized while the working efficiency of the engineering personnel is greatly improved, and the uniform 6D pose data acquisition of the object is quickly completed.

[0084] Step S26: extracting a target image frame from the cut video, and determining the pose information corresponding to the target image frame by using a preset visual SLAM algorithm.

[0085] The more specific processing procedure about step S26 can refer to the corresponding content disclosed in the foregoing embodiments, and will not be described here in detail.

[0086] The embodiment proposes to use a fast search random number method for robot motion planning, and to use a preset video automatic editing script to cut the video segment in the target video for adjusting the pose of the robot, wherein the video editing process can use automatic editing or semi-automatic editing with manual annotation. Finally, the target image frame is extracted from the edited video, and the preset visual SLAM algorithm is used to determine the pose information corresponding to the target image frame. The method in the embodiment greatly reduces the repeated manual operation required in the 6D pose data set collection process of the object, and replaces the handheld mode with a robot, making the entire collection process more automated. The pose data extracted using the method of the present application can ensure good scene reconstruction effect, and at the same time greatly improve the work efficiency of the engineers, and can quickly obtain high-precision and high-reliability 6D pose data of any object in almost any scene background.

[0087] Referring to Figure 6 The embodiment of the present application discloses a pose data acquisition device, which can specifically include:

[0088] The sampling viewpoint determination module 11 is configured to determine a target sampling viewpoint from a spherical surface with the center of the target object as the spherical center and a preset sampling distance as the radius;

[0089] The motion trajectory determination module 12 is configured to determine a target motion trajectory of a robot carrying a camera at the end based on the target sampling viewpoint and using a preset trajectory planning method;

[0090] The target video recording module 13 is configured to control the robot to move in the target motion trajectory, and record a video of the target object using the camera carried at the end of the robot, to obtain a target video; wherein a ray emitted by the camera in a forward direction, which is perpendicular to the camera pixel plane and passes through the optical center, always passes through the center of the target object.

[0091] The pose data determination module 14 is configured to extract a target image frame from the target video, and determine pose information corresponding to the target image frame using a preset visual SLAM algorithm.

[0092] In the present application, first, the target sampling viewpoint is determined on the spherical surface with the center of the target object as the spherical center and the preset sampling distance as the radius; based on the target sampling viewpoint, the target motion trajectory of the mechanical arm carrying the camera at the end is determined by using a preset trajectory planning method; the mechanical arm is controlled to move in the target motion trajectory, and the target object is video recorded by using the camera carried at the end of the mechanical arm to obtain a target video; wherein the ray emitted by the camera in front and perpendicular to the camera pixel plane and passing through the optical center always passes through the center of the target object; the target image frame is extracted from the target video, and the corresponding pose information of the target image frame is determined by using a preset visual SLAM algorithm. In this way, the present application uses the independently designed "spherical surface sampling" method to make the mechanical arm drive the camera at the end to move according to the specifically planned spherical surface trajectory, and ensures that the camera is always aligned to the object to be collected during the movement, and finally extracts the pose data of the target object from the target video shot by the camera. Since the pose data collected by the mechanical arm spherical surface sampling method in the present application has the characteristic of uniform distribution, the pose information of the target object extracted finally is more complete, which can provide great help for the development, test, verification and improvement of 6D pose estimation algorithm.

[0093] Further, the present application also discloses an electronic device, Figure 7 The electronic device 20 structure diagram shown according to the exemplary embodiments, the contents in the figure cannot be considered as any limitation on the use range of the present application.

[0094] Figure 7 The electronic device 20 structure diagram provided by the present application embodiment. The electronic device 20, specifically can include: at least one processor 21, at least one memory 22, power supply 23, display screen 24, input output interface 25, communication interface 26 and communication bus 27. Among them, the memory 22 is used to store computer programs, the computer programs are loaded and executed by the processor 21, to realize the related steps in the pose data acquisition method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the present embodiment can be an electronic computer.

[0095] In the present application, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 26 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol followed is any communication protocol applicable to the technical solution of the present application, which is not limited here; the input output interface 25 is used to obtain external input data or output data to the outside world, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0096] In addition, the memory 22 can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc. as a carrier for storing resources, and the resources stored thereon can include an operating system 221, a computer program 222, virtual machine data 223, etc. The virtual machine data 223 can include various data. The storage mode can be temporary storage or permanent storage.

[0097] The operating system 221 is used to manage and control various hardware devices on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the pose data acquisition method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0098] Further, the present application also discloses a computer readable storage medium, which includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk, or any other form of storage medium known in the technical field. The computer program is executed by a processor to implement the pose data acquisition method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be described here.

[0099] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part. The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the composition and steps of each example have been described in general in the above description. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0100] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC.

[0101] Finally, it should be noted that the terms "first" and "second", and the like, are used herein only to distinguish one entity or action from another, but do not necessarily require or imply these entities or actions are in any way mutually exclusive or directional. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0102] The pose data acquisition method, device, equipment and storage medium provided by the present application are introduced in detail above, and the principles and implementation manners of the present application are described by applying specific examples in the present article. The above description of the embodiments is only for helping to understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description of the present application should not be understood as the limitation of the present application.

Claims

1. A method for acquiring pose data, characterized in that, include: The target sampling viewpoint is determined from a sphere with the center of the target object as the center and a preset sampling distance as the radius. Based on the target sampling viewpoint, the target motion trajectory of the robotic arm with a camera at the end is determined using a preset trajectory planning method. The robotic arm is controlled to move along the target trajectory, and the camera carried at the end of the robotic arm is used to record video of the target object to obtain target video; wherein, the ray emitted forward by the camera, perpendicular to the camera pixel plane and passing through the optical center, always passes through the center of the target object; The target image frame is extracted from the target video, and the pose information corresponding to the target image frame is determined using a preset visual SLAM algorithm.

2. The pose data acquisition method according to claim 1, characterized in that, The step of determining the target sampling viewpoint from a sphere with the center of the target object as the center and a preset sampling distance as the radius includes: Based on preset horizontal and vertical sampling density values, the target sampling viewpoint is determined on a sphere with the center of the target object as the center and a preset sampling distance as the radius.

3. The pose data acquisition method according to claim 1, characterized in that, The step of determining the target motion trajectory of the robotic arm carrying a camera at its end based on the target sampling viewpoint and using a preset trajectory planning method includes: Based on the target sampling viewpoint, the target motion trajectory of the robotic arm with a camera at the end is determined using a fast search random number method.

4. The pose data acquisition method according to claim 1, characterized in that, The step of extracting target image frames from the target video and determining the pose information corresponding to the target image frames using a preset visual SLAM algorithm includes: The video clips used to adjust the pose of the robotic arm in the target video are cut out using a preset automatic video editing script to generate an edited video; The target image frame is extracted from the edited video, and the pose information corresponding to the target image frame is determined using a preset visual SLAM algorithm.

5. The pose data acquisition method according to claim 4, characterized in that, The step of using a preset video automatic editing script to cut out video segments from the target video used to adjust the pose of the robotic arm includes: A preset visual detection algorithm is used to determine the key points of the video frame used to adjust the pose of the robotic arm, and the video segment to be edited is determined based on the key points of the video frame. Alternatively, it may receive a setting instruction for setting key points of video frames in the target video for adjusting the pose of the robotic arm, and determine the video segment to be edited based on the setting instruction.

6. The pose data acquisition method according to claim 1, characterized in that, Before determining the target motion trajectory of the robotic arm carrying a camera at its end based on the target sampling viewpoint and using a preset trajectory planning method, the method further includes: The coordinate transformation relationship between the robotic arm and the camera carried at the end of the robotic arm can be obtained by using a hand-eye calibration method; or, the coordinate transformation relationship between the robotic arm and the camera carried at the end of the robotic arm can be determined by using SolidWorks macro tools. Based on the coordinate transformation relationship, the target setting parameters for setting the robotic arm are determined, and the target setting parameters are set for the robotic arm.

7. The pose data acquisition method according to any one of claims 1 to 6, characterized in that, The method of determining the target motion trajectory of the robotic arm with a camera at its end effector using a preset trajectory planning method includes: The target motion trajectory of the robotic arm with a camera at its end effector and the operating speed of the robotic arm are determined by using a preset trajectory planning method. Accordingly, controlling the robotic arm to move along the target trajectory and using the camera carried at the end of the robotic arm to record video of the target object to obtain target video includes: Control the camera to start video recording, and start the robotic arm to move at the running speed according to the target motion trajectory; When the robotic arm moves to the target sampling viewpoint, the robotic arm is controlled to pause its movement. The rotating device on the robotic arm used to rotate the camera is controlled to rotate the camera 360 degrees. When the rotating device rotates back to its initial position, the robotic arm is controlled to continue moving at the running speed according to the target motion trajectory. When the robotic arm moves to the next target sampling viewpoint, the process jumps to the step of controlling the robotic arm to pause its movement until the robotic arm finishes moving along the target motion trajectory. Control the camera to turn off recording in order to obtain the target video recorded by the camera.

8. A pose data acquisition device, characterized in that, include: The sampling viewpoint determination module is used to determine the target sampling viewpoint from a sphere with the center of the target object as the center and a preset sampling distance as the radius. The motion trajectory determination module is used to determine the target motion trajectory of the robotic arm with a camera at its end based on the target sampling viewpoint and using a preset trajectory planning method. The target video recording module is used to control the robotic arm to move along the target motion trajectory and to use the camera carried at the end of the robotic arm to record video of the target object to obtain target video; wherein, the ray emitted forward by the camera, perpendicular to the camera pixel plane and passing through the optical center, always passes through the center of the target object; The pose data determination module is used to extract target image frames from the target video and determine the pose information corresponding to the target image frames using a preset visual SLAM algorithm.

9. An electronic device, characterized in that, It includes a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the pose data acquisition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the pose data acquisition method as described in any one of claims 1 to 7.

Images