Pose estimation model training methods, devices, terminal equipment, and storage media

Abstract

This application relates to the field of deep learning and discloses a method, apparatus, terminal device, and storage medium for training a pose estimation model. The method includes: generating a hybrid initial pose set associated with the ground truth pose of a target object in a real image, the hybrid initial pose set including multiple local initial poses and multiple global initial poses; generating a rendered image set of the target object based on the 3D model of the target object and the initial pose set, the rendered image set including color images and depth images rendered by the target object under each initial pose; pairing the rendered image set and the real image and inputting them into the pose estimation model, using the pose error between each initial pose and the ground truth pose as a supervision signal for model training, the pose estimation model being used to predict the 3D position and 3D pose angle of the target object. This method can improve the convergence speed, robustness, and pose estimation accuracy of the pose estimation model.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a pose estimation model training method, apparatus, terminal device and storage medium. Background Technology

[0002] In industrial applications, when robots perform common tasks such as sorting, grasping, or assembly, they need to acquire images of target objects using their own vision sensors. Then, they use 6D pose estimation methods to identify the 6D pose of the target object based on the acquired images, including the target object's three-dimensional position (x, y, z) and three-dimensional attitude angles (roll, pitch, yaw) or rotation matrix R, thereby enabling subsequent high-precision task operations.

[0003] In deep learning-based 6D pose estimation methods, based on whether a CAD model of the object is available, deep learning model estimation methods can be divided into CAD model-based 6D pose estimation and CAD model-free 6D pose estimation. Generally, CAD model-based pose estimation methods offer higher accuracy and can be applied in precision-critical fields such as industrial robot grasping, assembly, and navigation. However, training a CAD model-based 6D pose estimation model typically requires constructing several initial poses, and then generating a rendered image based on these initial poses and the object's CAD model. The sampling and generation methods of the initial poses significantly impact the accuracy and convergence speed of the pose estimation model, and current initial pose sampling methods cannot simultaneously meet the demands for both high estimation accuracy and fast convergence speed. Summary of the Invention

[0004] In view of this, embodiments of this application provide a pose estimation model training method, apparatus, terminal device, and storage medium.

[0005] In a first aspect, embodiments of this application provide a pose estimation model training method, including: Based on the ground truth pose of the target object in the real image, a hybrid initial pose set associated with the ground truth pose is generated. The hybrid initial pose set includes multiple local initial poses and multiple global initial poses. Based on the 3D model of the target object and the initial pose set, a set of rendered images of the target object is generated, the set of rendered images including color images and depth images of the target object rendered in each initial pose; The rendered image set and the real image are paired and input into the pose estimation model, and the pose error between each initial pose and the ground truth pose is used as a supervision signal for model training. The pose estimation model is used to predict the three-dimensional position and three-dimensional pose angle of the target object.

[0006] Secondly, embodiments of this application provide a pose estimation model training device, comprising: The initial pose generation module is used to generate a hybrid initial pose set associated with the ground truth pose of the target object in the real image as a reference. The hybrid initial pose set includes multiple local initial poses and multiple global initial poses. The image rendering module is used to generate a set of rendered images of the target object based on the three-dimensional model of the target object and the initial pose set. The set of rendered images includes a color image and a depth image of the target object rendered in each initial pose. The model training module is used to pair the rendered image set and the real image and input them into the pose estimation model, and use the pose error between each initial pose and the true pose as a supervision signal for model training. The pose estimation model is used to predict the three-dimensional position and three-dimensional pose angle of the target object.

[0007] Thirdly, embodiments of this application provide a terminal device, the terminal device including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the pose estimation model training method of the above embodiments.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed on a processor, implements the pose estimation model training method of the above embodiments.

[0009] The embodiments of this application have the following beneficial effects: This application proposes a pose estimation model training method. It generates an initial pose using a hybrid global sampling and local sampling method, which is then used to train the pose estimation model. Because the initial pose is trained using a globally sampled pose, the pose estimation model has a stronger global search capability, improving pose estimation accuracy. Therefore, it has better recovery capability for initial poses with large deviations, can adapt to various pose scenarios, and is more robust. Simultaneously, because the initial pose is trained using a locally sampled pose, the pose estimation model can better fit initial poses with small pose differences and assist the model in learning poses with larger deviations, thereby improving the overall model convergence speed and saving model training time. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This paper illustrates the application architecture diagram of the pose estimation model training method according to an embodiment of this application; Figure 2 A schematic diagram of the structure of a terminal device according to an embodiment of this application is shown; Figure 3 A first flowchart of the pose estimation model training method according to an embodiment of this application is shown; Figure 4 A second flowchart of the pose estimation model training method according to an embodiment of this application is shown; Figure 5 A schematic diagram of sampling on an icosahedron according to an embodiment of this application is shown; Figure 6 An example of rendered image data and real image data according to an embodiment of this application is shown; Figure 7 A schematic diagram of a pose estimation model training device according to an embodiment of this application is shown. Detailed Implementation

[0012] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0013] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0014] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0015] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0016] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0017] When training a deep learning-based 6D pose estimation model, several initial poses need to be constructed. Then, a rendered image is generated based on these initial poses and the object's CAD model. If the distribution of the generated initial poses differs significantly from the actual pose distribution of the object during inference, it will affect the pose estimation accuracy of the 6D pose estimation deep learning model. Currently, most pose estimation models construct initial poses in two ways during training: one is to obtain several initial poses by randomly perturbing the pose distribution near the ground truth of the training data; the other is to generate several initial poses based on global sampling. In practical applications, both methods have some problems and cannot meet certain specific application scenarios. For example, although the first method has a faster training convergence speed, because the pose distribution of the training data is close to the ground truth, the model's global search ability is affected. Consequently, during the inference stage, if the initial pose is far from the ground truth, the pose estimation model will struggle to obtain accurate pose estimation results. While the second method, using global sampling, optimizes the model's global search ability, it reduces the local accuracy of the pose estimation model and the convergence speed during training.

[0018] To this end, this application proposes a pose estimation model training method, a pose estimation model training device, a terminal device, and a computer-readable storage medium. Regarding the pose estimation model training method, it mainly adopts a dual-source initial pose supervision approach, that is, by using the ground truth pose of the same object as both the local perturbation center and the global uniform sampling reference origin, corresponding local initial poses and global initial poses are constructed, which are then used to generate rendering images with the CAD model of the object to supervise the training of the pose estimation model. In this way, the local convergence ability and global stability of the pose estimation model can be optimized simultaneously during training, thereby making the pose estimation model more robust and more accurate in the inference stage.

[0019] The following describes exemplary applications of the terminal devices provided in the embodiments of this application. These terminal devices can be implemented as various types of application-end devices (such as edge computing devices, mobile devices, etc.) or servers. In some embodiments, the pose estimation model training method and the trained pose estimation model of the embodiments of this application can be directly run on application-end devices such as edge computing devices and mobile devices. It is understood that to directly run the pose estimation model training method of the embodiments of this application, these application-end devices should generally have corresponding computing power support. Optionally, in some other embodiments, the pose estimation model training method of the embodiments of this application can also run on a server. That is, the server is used to train the pose estimation model, and then the trained pose estimation model is deployed locally on the server. Alternatively, it can be sent to specific application-end devices such as the aforementioned edge computing devices or mobile devices for deployment, so that the server itself or these application-end devices have pose estimation capabilities.

[0020] For example, Figure 1 This is a schematic diagram of the application architecture of the pose estimation model training method provided in the embodiments of this application. Exemplarily, Figure 1 This application involves a server 200, a network 300, and an application device 400. The method in this embodiment runs on the server 200, and the application device 400 connects to the server 200 through the network 300. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. In this embodiment, an automated assembly robot is used as an example of the application device 400. After the server 200 trains the pose estimation model using the pose estimation model training method proposed in this embodiment, it sends its network 300 to the application device 400 (i.e., the automated assembly robot) for deployment. After successful deployment, the automated assembly robot can collect an image containing the target object, process the image by calling the pose estimation model, and output the pose estimation result of the target object.

[0021] In some embodiments, the server (such as server 200 above) may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0022] In some embodiments, if the terminal device is an application-end device, it may include, but is not limited to, edge computing devices and mobile devices with at least moderate computing power that support lightweight deep learning model inference, and may involve various fields such as industry, healthcare, and consumer electronics. For example, edge computing devices may include, but are not limited to, industrial control computers, logistics sorting robots, assembly robots, and AGVs; mobile devices may include, but are not limited to, various types of portable or mobile devices such as laptops, tablets, smartphones, in-vehicle terminals, and AR / VR devices.

[0023] Figure 2 This is a schematic diagram of the structure of the terminal device 100 provided in the embodiments of this application. The terminal device 100 may be the application terminal device or server, as described above. Figure 2 As shown, the terminal device 100 can be, but is not limited to, various types of application terminal devices involving pose estimation, such as robotic arms and robots. It is especially suitable for industrial and service scenarios such as robot grasping, navigation, or assembly. Taking robots as an example, the terminal device 100 can be humanoid robots, wheeled robots, etc., and its form is not limited.

[0024] Examplely, the terminal device 100 includes a processor 110, a memory 120, and a bus 130. The memory 120 stores a computer program. When the terminal device 100 is running, the processor 110 communicates with the memory 120 via the bus 130 to run the computer program, thereby enabling the terminal device to perform the functions of the pose estimation model training method or the various modules in the pose estimation model training device 10 described above.

[0025] Further optional, if it is an application device such as an industrial robot, the application device usually also includes a visual perception unit (such as a camera or webcam), wherein the visual perception device can be used to acquire images containing the target object in real time, and then use the 6D pose estimation model trained by the method of the above embodiment to obtain the 6D pose estimation of the target object, thereby driving the application device (such as a robot) to complete corresponding operations, such as performing tasks such as grasping, assembly or autonomous navigation.

[0026] Processor 110 can be an integrated circuit chip with signal processing capabilities. Processor 110 can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0027] The memory 120 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 120 is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0028] As mentioned above, the terminal device implementing the pose estimation model training method of the embodiments of this application can be an application device, a server, or a combination of both. The following description uses the application device as an example to illustrate the pose estimation model training method provided in the embodiments of this application.

[0029] Figure 3 A flowchart illustrating a pose estimation model training method according to an embodiment of this application is shown. Exemplarily, the pose estimation model training method includes steps S110-S150: S110, using the ground truth pose of the target object in the real image as a reference, generate a set of hybrid initial poses associated with the ground truth poses. The set of hybrid initial poses includes multiple local initial poses and multiple global initial poses.

[0030] For example, for each real image used as a training sample, which typically refers to an actual image acquired by a visual sensor (such as a camera), for example, the actual image may include a color image (such as an RGB image) and a depth image, the ground truth pose of the target object (such as a ground truth 6D pose, including three-dimensional position and three-dimensional pose angle) is obtained from the real image; then, based on the ground truth pose, i.e. around the ground truth pose, a series of initial poses corresponding to the ground truth pose are generated, thereby forming a set of hybrid initial poses.

[0031] It is worth noting that this application employs both local and global sampling methods during sampling. Therefore, the series of initial poses obtained above are actually composed of multiple local initial poses generated by the local sampling method and multiple global initial poses generated by the global sampling method.

[0032] It can be understood that the aforementioned ground truth pose association means that there is a one-to-one correspondence between the mixed initial pose set and the ground truth pose. That is, for a ground truth pose in a real image, a corresponding mixed initial pose set can be generated through the above steps. Then, by pairing the ground truth pose with these mixed initial poses, they are input as a set of sample data into the pose estimation model to be trained for batch training. When there are multiple real images and corresponding real poses, different batches of training can be performed.

[0033] In some embodiments, such as Figure 4 As shown, the mixed initial pose set in step S110 above can be generated by the following steps: S210: Add random noise perturbation to the true pose of the target object in the real image multiple times to generate multiple local initial poses.

[0034] The random noise perturbation mentioned above can be, but is not limited to, using Gaussian noise or similar scrambling techniques. Exemplarily, by applying a random perturbation to the neighborhood of the true pose—that is, adding random Gaussian noise to the true pose—a new hypothetical initial pose can be obtained. Thus, through N random noise perturbation processes, N local initial poses can be obtained. The number of samples N for the local initial poses is typically an integer greater than 0. In engineering practice, considering the balance between performance and convergence time, N can be, for example, 10, 12, or 15. Of course, these are just some feasible examples; the specific setting can be determined according to actual needs, and no single limitation is made here.

[0035] For example, in one implementation, if described by an expression, we have: applying a small-angle Lie algebraic perturbation δξ to the rotation matrix R in the true pose yields the perturbed rotation matrix R′=exp(δξ) R; and applying an isotropic Gaussian noise perturbation δt to the translation t in the true pose yields the perturbed translation t′=t+δt; combining the two vibration parts (R′,t′) constitutes a local initial pose.

[0036] S230: A set of candidate target points is obtained by uniformly sampling on a regular polyhedron centered on the target object, and multiple global initial poses are determined from the set of candidate target points.

[0037] In this embodiment, the target object is used as the center of a sphere, and the required global initial pose is generated by uniformly sampling the spherical surface using the geometry of a regular polyhedron. This can be understood as placing a camera at different positions on the regular polyhedron, each facing the center of the target object, to capture images of the target object from different angles. Taking robot assembly as an example, the camera mounted on the robot is used to capture images of the assembled workpiece from different perspectives to obtain the workpiece's pose information in three-dimensional space, thereby achieving high-precision assembly tasks.

[0038] For example, in some embodiments, the process of generating the global initial pose in step S230 above includes: From the regular polyhedron centered on the target object, select all vertices and the midpoint of each edge as candidate points to form a set of candidate target points. Then, randomly select a preset number of target points from the set of candidate target points and determine the spatial coordinates of each target point as multiple global initial poses.

[0039] It is worth noting that, in addition to selecting all vertices on the regular polyhedron as the assumed initial pose, interpolation is also performed. That is, the midpoint of each edge of the regular polyhedron is selected as an additional candidate point. In this way, on the one hand, the number of candidate points can be increased, and on the other hand, the coverage of the global initial pose in the pose space can be further improved by combining sampling of "vertex + edge midpoint".

[0040] For the aforementioned set of candidate target points, as an optional approach, if the number of candidate points exceeds the required number of global initial poses, a portion of the target points can be randomly selected. By randomly selecting again from all globally uniformly sampled data, the diversity of model training can be further guaranteed. Alternatively, an equal number of candidate points can be selected from regular polyhedra according to the required number of global initial poses. In this case, the positions of the candidate points can be selected at any time, and there is no limitation here.

[0041] As an optional implementation, the number of local initial poses and the number of global initial poses can be the same. In practice, optionally, the ratio and total number of these two types of initial poses can be adjusted according to the balance between the accuracy performance and convergence time of the pose estimation model, etc., and no unique limitation is made here.

[0042] The aforementioned regular polyhedra may include, but are not limited to, regular tetrahedrons, regular hexahedrons, regular octahedrons, regular dodecahedrons, or regular icosahedrons. Preferably, a regular icosahedron can be used because it is the regular polyhedron with the most faces, and compared to regular octahedrons and regular dodecahedrons, it is closer to a sphere, thus allowing for the sampling of a larger number of poses, resulting in higher sampling efficiency and more uniform sampling.

[0043] For example, such as Figure 5 As shown, taking a regular icosahedron as an example, the target object is taken as the center of the icosahedron, and any vertex of the icosahedron is assumed to be a possible initial pose. By selecting all vertices of the icosahedron, 12 candidate points can be obtained. Then, the position of the midpoint of each edge of the icosahedron is calculated as a candidate point for interpolation, as shown below. Figure 3 As shown, the two green dots are the two vertices of a regular icosahedron, and the red dots are the midpoints of the edges containing these two vertices. By calculating the midpoints of all edges of the icosahedron, 30 candidate points can be obtained; adding the 12 vertices of the icosahedron, a candidate target point set of 42 points can be obtained. Furthermore, to ensure the diversity of training, 12 points are randomly selected from these 42 candidate points as the global initial pose for construction.

[0044] S250, the generated multiple local initial poses and multiple global initial poses are merged to obtain a set of mixed initial poses associated with the ground truth poses.

[0045] In this embodiment, the two types of initial poses are merged to form a hybrid initial pose set for subsequent pose estimation model training. It can be understood that the aforementioned local initial poses characterize the local distribution of the pose space, while the global initial poses characterize the global connectivity of the pose space. Since the global pose distribution covers the 6D pose space, the pose estimation model trained using globally sampled initial poses will have stronger global search capabilities, thus exhibiting better recovery capabilities for initial poses with large deviations and adapting to various pose scenarios. Simultaneously, because the model training uses local initial poses, the pose estimation model can better fit initial poses with smaller pose differences, while also assisting the model in learning poses with larger deviations, improving the overall model convergence speed and saving model training time.

[0046] S130, Based on the three-dimensional model of the target object and the initial pose set, generate a set of rendered images of the target object, which includes color images and depth images of the target object rendered in each initial pose.

[0047] Exemplarily, for each initial pose (local or global) in the initial pose set, the 3D model of the target object is projected onto a 2D image plane using the camera intrinsic matrix and the current initial pose, and rendered to generate a color image and a depth image of the target object rendered in the current initial pose, thus obtaining a rendered image set. This rendered image set consists of multiple rendered image data obtained from multiple initial poses. A rendered image data set has the same image type as the real image data; that is, if the real image data includes both a real RGB color image and a depth image acquired from a certain viewpoint, then a rendered image data set will correspondingly include the rendered RGB color image and the depth image obtained in the corresponding initial pose.

[0048] S150, the rendered image set and the real image are paired and input into the pose estimation model, and the pose error between each initial pose and the true pose is used as a supervision signal for model training. The pose estimation model is used to predict the three-dimensional position and three-dimensional pose angle of the target object (i.e., 6D pose estimation model).

[0049] Exemplarily, during training, the obtained rendered image data is paired with the acquired real image data and input as a set of training samples into the pose estimation model. Then, the pose error between each initial pose and the ground truth pose is used as a supervision signal to train the pose estimation model. It can be understood that the pose estimation model of this application is mainly aimed at 6D pose estimation, but the approach of using hybrid sampling to simultaneously ensure the model's convergence speed and estimation accuracy can also be applied to other scenarios requiring pose estimation model training and pose estimation.

[0050] When pairing inputs, for example, the rendered RGB color image can be channel-wise concatenated with the acquired real RGB color image, and the rendered depth image can be channel-wise concatenated with the acquired real depth image, or feature-wise aligned (e.g., pixel-level feature alignment). Figure 6As shown, taking the aforementioned workpiece as an example, the four images from left to right are, in order, an aligned RGB image rendered from a CAD model of a certain workpiece, a captured real RGB image, a depth image rendered from the CAD model, and a captured real depth image. It can be understood that RGB-Depth alignment can effectively solve the offset problem between the rendered and real domains. Furthermore, since all local initial poses and global initial poses participate in the same round of model parameter updates during training, this pose estimation model can simultaneously learn local convergence and global pose recovery capabilities.

[0051] In one embodiment, the above-described model training process includes: using the pose error between each initial pose and the ground truth pose in the rendered image set as the optimization objective; and simultaneously updating the parameters of the pose estimation model using gradient descent during the optimization process until training is complete, resulting in a trained pose estimation model. Gradient descent is used to minimize the objective function (such as a loss function), which iteratively updates the parameters along the inverse direction of the objective function's gradient to gradually approach the local or global minimum.

[0052] In designing the optimization loss function, this application considers both local and global error terms of the pose. Optionally, an optimization function aimed at minimizing the pose error may be used. For example, assuming there are a total of 24 initial local and global poses in the initial pose set, matrix calculations are performed between these 24 initial poses and the true pose to obtain 24 error values. These error values ​​serve as the target to be optimized in the pose estimation model. These error values ​​are the weighted sum of the pose rotation error and the position translation error. As an optional approach, the weighting coefficients can be dynamically adjusted according to the training iteration stages. For example, the position translation error can be emphasized in the initial stage, while the pose rotation error weight can be increased in later stages.

[0053] It is understandable that injecting local and global poses simultaneously and jointly optimizing the same training sample can lead to gradient conflicts if the sampling distributions of the two are significantly different (Gaussian concentration vs. spherical dispersion). This application can effectively solve the above-mentioned technical obstacles by designing a unified error target.

[0054] The pose estimation model training method in this application employs a hybrid sampling method for the initial pose. This involves a portion of the initial pose based on perturbation sampling near the true pose in the training data (local sampling), and another portion based on uniform sampling of the sphere of a regular polyhedron or other globally distributed data (global sampling). This allows the model to learn data distributions whose initial poses are close to the true pose, while also estimating initial poses that deviate significantly from the true pose. The initial pose obtained through this hybrid sampling method simultaneously optimizes the local convergence capability and global stability of the pose estimation model during training. This results in stronger robustness and higher accuracy for the pose estimation model during the inference phase.

[0055] It is understood that the pose estimation model trained by the method in the embodiments of this application can be deployed in the control system of application-end devices such as robots, robotic arms, and engineering equipment, so as to perform various tasks such as grasping, assembly, or autonomous navigation. Taking a robot as an example, a robot integrating a 6D pose estimation model acquires image information (such as RGB color images and depth images) containing the target object during the execution of a task through its own installed vision sensors (such as cameras), and then inputs it into the 6D pose estimation model to identify the 6D pose of the target object, including the three-dimensional position (x, y, z or translation vector t) and three-dimensional attitude angles (roll, pitch, yaw) or rotation matrix R of the target object, thereby realizing subsequent high-precision task operations.

[0056] Figure 7 A schematic diagram of a pose estimation model training device 500 according to an embodiment of this application is shown. Exemplarily, the pose estimation model training device 500 includes: The initial pose generation module 510 is used to generate a hybrid initial pose set associated with the ground truth pose of the target object in the real image as a reference. The hybrid initial pose set includes multiple local initial poses and multiple global initial poses. The image rendering module 530 is used to generate a set of rendered images of the target object based on the three-dimensional model of the target object and the initial pose set. The set of rendered images includes a color image and a depth image of the target object rendered in each initial pose. The model training module 550 is used to pair the rendered image set and the real image into the pose estimation model, and to train the model using the pose error between each initial pose and the true pose as a supervision signal. The pose estimation model is used to predict the three-dimensional position and three-dimensional pose angle of the target object.

[0057] Furthermore, the initial pose generation module 510 includes a local pose generation submodule, a global pose generation submodule, and a merging processing submodule. The local pose generation submodule is used to add random noise perturbation to the ground truth pose of the target object in the real image multiple times to generate multiple local initial poses. The random noise perturbation is scrambled using Gaussian noise. The global pose generation submodule is used to uniformly sample a set of candidate target points on a regular polyhedron centered on the target object, and determine multiple global initial poses from the set of candidate target points; wherein, each vertex of the regular polyhedron and the midpoint of each edge are set as candidate points in the set of candidate target points; The merging processing submodule is used to merge the multiple local initial poses and the multiple global initial poses, and output the mixed initial pose set associated with the true pose.

[0058] As an optional approach, the global pose generation submodule is specifically used to select all vertices and the midpoint of each edge from the regular polyhedron centered on the target object as all candidate points to form a set of candidate target points, and randomly select a preset number of target points from the set of candidate target points and determine the spatial coordinates of each target point as multiple global initial poses.

[0059] As an alternative, the regular polyhedron is a regular tetrahedron, a regular hexahedron, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0060] As an optional approach, the number of local initial poses is the same as the number of global initial poses.

[0061] Furthermore, the image rendering module 530 is specifically used to, for each initial pose in the initial pose set, project the three-dimensional model of the target object onto a two-dimensional image plane through the camera intrinsic parameter matrix and the current initial pose and render it, thereby generating a color image and a depth image of the target object rendered under the current initial pose, and thus obtaining a rendered image set.

[0062] Furthermore, when training the model, the model training module 550 specifically uses the pose error between each initial pose and the ground truth pose in the rendered image set as the optimization target, and uses the gradient descent method to update the parameters of the pose estimation model to obtain the trained pose estimation model.

[0063] It is understood that the apparatus in this embodiment corresponds to the pose estimation model training method in the above embodiments, and the options in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0064] In some embodiments, the pose estimation model training device 500 provided in this application can be implemented in software, specifically in the form of programs and plug-ins stored in the memory 120. The composition of each module in the pose estimation model training device 500 is logical, and therefore can be arbitrarily combined or further divided according to the functions implemented. The functions of each module will be described below.

[0065] This application also provides a computer-readable storage medium for storing the computer program used in the terminal device 100 described above. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0066] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0067] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0068] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0069] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for training a pose estimation model, characterized in that, include: Based on the ground truth pose of the target object in the real image, a hybrid initial pose set associated with the ground truth pose is generated. The hybrid initial pose set includes multiple local initial poses and multiple global initial poses. Based on the 3D model of the target object and the initial pose set, a set of rendered images of the target object is generated, the set of rendered images including color images and depth images of the target object rendered in each initial pose; The rendered image set and the real image are paired and input into the pose estimation model, and the pose error between each initial pose and the ground truth pose is used as a supervision signal for model training. The pose estimation model is used to predict the three-dimensional position and three-dimensional pose angle of the target object.

2. The pose estimation model training method according to claim 1, characterized in that, The step of generating a hybrid initial pose set associated with the ground truth pose of the target object in the real image as a reference includes: Random noise perturbation is added multiple times to the ground truth pose of the target object in the real image to generate multiple local initial poses; wherein, the random noise perturbation is scrambled using Gaussian noise. A set of candidate target points is obtained by uniformly sampling on a regular polyhedron centered on the target object, and multiple global initial poses are determined from the set of candidate target points; wherein, each vertex of the regular polyhedron and the midpoint of each edge are set as candidate points in the set of candidate target points; The multiple local initial poses and the multiple global initial poses are merged to obtain the hybrid initial pose set associated with the truth pose.

3. The pose estimation model training method according to claim 2, characterized in that, The process of uniformly sampling a candidate target point set on a regular polyhedron centered on the target object, and determining multiple global initial poses from the candidate target point set, includes: From the regular polyhedron centered on the target object, all vertices and the midpoint of each edge are selected as candidate points to form a set of candidate target points. A preset number of target points are randomly selected from the set of candidate target points, and the spatial coordinates of each target point are determined as multiple global initial poses.

4. The pose estimation model training method according to claim 2 or 3, characterized in that, The regular polyhedron is a regular tetrahedron, a regular hexahedron, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

5. The pose estimation model training method according to claim 1, characterized in that, The number of local initial poses is the same as the number of global initial poses.

6. The pose estimation model training method according to claim 1, characterized in that, The step of generating a set of rendered images for the target object based on the 3D model of the target object and the initial pose set includes: For each initial pose in the initial pose set, the 3D model of the target object is projected onto a 2D image plane using the camera intrinsic matrix and the current initial pose and rendered to generate a color image and a depth image of the target object rendered under the current initial pose, thereby obtaining a set of rendered images.

7. The pose estimation model training method according to claim 1, characterized in that, The step of using the pose error between each initial pose and the true pose as a supervision signal for model training includes: The pose error between each initial pose and the ground truth pose in the rendered image set is used as the optimization objective, and the pose estimation model is updated using the gradient descent method to obtain a trained pose estimation model.

8. A pose estimation model training device, characterized in that, include: The initial pose generation module is used to generate a hybrid initial pose set associated with the ground truth pose of the target object in the real image as a reference. The hybrid initial pose set includes multiple local initial poses and multiple global initial poses. The image rendering module is used to generate a set of rendered images of the target object based on the three-dimensional model of the target object and the initial pose set. The set of rendered images includes a color image and a depth image of the target object rendered in each initial pose. The model training module is used to pair the rendered image set and the real image and input them into the pose estimation model, and use the pose error between each initial pose and the true pose as a supervision signal for model training. The pose estimation model is used to predict the three-dimensional position and three-dimensional pose angle of the target object.

9. A terminal device, characterized in that, The terminal device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the pose estimation model training method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed on a processor, implements the pose estimation model training method according to any one of claims 1-7.

Images