Six-degree-of-freedom object attitude estimation method and system based on differential rendering

Abstract

The invention discloses a six-degree-of-freedom object attitude estimation method and system based on differential rendering, and the method comprises the steps: carrying out the rough registration operation of six-degree-of-freedom pose estimation of a three-dimensional model of an object, and obtaining a first attitude of the object; creating a plurality of optimization instances with random initial learning rates according to the first posture; performing attitude optimization of learning rate attenuation on each optimization instance by adopting a differential renderer-based gradient descent algorithm and a multi-modal loss function to obtain a plurality of attitude results; taking the attitude result with the lowest value of the multi-modal loss function as a final attitude estimation result; according to the embodiment of the invention, by adopting a differential rendering mode, 6-DoF object attitude estimation without training data is realized, a high-precision attitude estimation result can be provided, meanwhile, better generalization ability and real-time performance are achieved, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and system for estimating the posture of a six-degree-of-freedom object based on differential rendering. Background Art

[0002] With the development of technology, there is an increasing demand for accurate six-degree-of-freedom (6-DoF) attitude estimation of objects in advanced fields such as robotics, augmented reality (AR) / mixed reality (MR), and automatic navigation. 6-DoF attitude estimation refers to determining the position and orientation of an object in three-dimensional space, which involves the three-dimensional translation and three-dimensional rotation of the object. The precise implementation of this technology is crucial to achieving precise control of robots, seamless integration of virtual information with the real world, and safe navigation of autonomous driving systems.

[0003] In related technologies, the pose estimation method based on template matching is to generate as many templates as possible, each template represents a different rotation pose, and calculate the similarity with the original image. The color gradient of the image is used to resist the influence of lighting and noise, and the depth information is introduced to construct several surface vertical vectors as feature template matching. However, due to the influence of lighting and occlusion in the actual environment, it is impossible to exhaust all situations as templates, the algorithm estimation accuracy is low, and a large number of matching templates will greatly increase the running time, so this type of algorithm performs poorly in this case.

[0004] In the related art, the deep learning method based on RGB images mainly estimates the 6D pose of a set of known objects from RGB-D images, processes two data sources separately, uses a new dense fusion network to extract pixel-level dense feature embedding, and estimates the pose from it. The 2D information in the embedding space is used to increase the information of each 3D point, and this new color space is used to estimate the 6D pose. Although this method uses the information of RGB and depth maps, it does not use other image information. At the same time, it directly integrates fine optimization steps into the network. The process is not intuitive enough, the interpretability is low, and it is difficult to confirm the optimization direction.

[0005] In addition, traditional techniques typically use the Jacobian matrix between image controls and pose controls to calculate the incremental pose required to minimize image-based errors. In this case, the neural network compares the rendered image of the 3D mesh with the target image to predict that the rendered image better matches the target model, and then updates the pose. Since these techniques rely on traditional non-rendering pipelines (for example, the 6DoF matrix obtained by direct regression is not necessarily reversible), the neural network must be trained offline on large datasets to learn to calculate pose updates. Once trained, the network is opaque, and it is difficult to improve performance without expensive and time-consuming retraining because it is impossible to understand why it performs well or fails. Summary of the invention

[0006] In order to solve at least one of the above problems, the main purpose of the embodiments of the present application is to propose a six-degree-of-freedom object pose estimation method and system based on differential rendering with high estimation accuracy and good interpretability, aiming to realize six-degree-of-freedom object pose estimation without the need for training data.

[0007] To achieve the above object, an embodiment of the present application proposes a six-degree-of-freedom object posture estimation method based on differential rendering, the method comprising:

[0008] Performing a coarse registration operation of six-degree-of-freedom pose estimation on the three-dimensional model of the object to obtain a first pose of the object;

[0009] According to the first posture, create several optimization instances with random initial learning rates;

[0010] Using a gradient descent algorithm based on a differentiable renderer and a multimodal loss function, performing posture optimization with a learning rate decay on each of the optimization instances, and obtaining a number of posture results;

[0011] The posture result with the lowest value of the multimodal loss function is taken as the final posture estimation result.

[0012] In some embodiments, performing a coarse registration operation of the three-dimensional model of the object for six-degree-of-freedom pose estimation to obtain the first pose of the object comprises one of the following steps:

[0013] By using a feature point matching algorithm, a six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object to obtain a first pose of the object;

[0014] Alternatively, a six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object through a template matching algorithm to obtain a first pose of the object;

[0015] Alternatively, a six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object through a deep learning model to obtain a first pose of the object.

[0016] In some embodiments, in the step of creating a plurality of optimization instances with random initial learning rates according to the first posture, the step of creating the optimization instance comprises the following steps:

[0017] Configure the upper and lower bounds of the learning rate;

[0018] Randomly determine a learning rate between the learning rate upper bound and the learning rate lower bound as the initial learning rate;

[0019] An optimization instance is created according to the first posture and the initial learning rate.

[0020] In some embodiments, the step of performing posture optimization with learning rate decay on each of the optimization instances using a gradient descent algorithm based on a differentiable renderer and a multimodal loss function to obtain a plurality of posture results comprises the following steps:

[0021] Acquiring an observation image of the three-dimensional model of the object;

[0022] Rendering a two-dimensional image of the three-dimensional model of the object according to the first posture, the three-dimensional model of the object and the camera intrinsic parameters by a differentiable renderer to obtain a rendered image;

[0023] According to the multimodal loss function, calculating the loss value and the loss function gradient between the rendered image and the observed image; wherein the loss function gradient is the gradient of the multimodal loss function relative to the first posture;

[0024] Performing posture updating according to the first posture, the current learning rate and the gradient of the loss function to obtain a second posture;

[0025] According to the learning rate attenuation factor, the current learning rate is updated, and the second posture is used as the first posture, and the step of rendering a two-dimensional image of the three-dimensional model of the object according to the first posture, the three-dimensional model of the object and the camera intrinsic parameters through a differentiable renderer to obtain a rendered image is repeatedly performed until the number of iterations reaches a preset iteration number threshold, and a posture result is obtained.

[0026] In some embodiments, the expression of the multimodal loss function is:

[0027] L=λ c L c +λ d L d +λ e L e ;

[0028]

[0029] Among them, L represents the multimodal loss function; L c is the RGB modality loss; L d is the depth modality loss; L e is the marginal mode loss; and λ c is the weight of the RGB mode; d is the weight of the depth modality; λ e is the weight of the edge mode; L EIoU represents EIoU loss; ⊙ represents element-by-element product; f function represents the minimum detection box of the image; I c is the observed RGB image; is to render RGB images; I d is the observed depth image; is the rendered depth image; I e is the observed edge map; is the rendered edge map; S represents the segmentation mask.

[0030] In some embodiments, the expression for performing posture update is:

[0031]

[0032] in, The second posture. is the first posture, α i is the current learning rate, is the gradient of the loss function.

[0033] In some embodiments, before the step of using a gradient descent algorithm based on a differentiable renderer and a multimodal loss function to perform posture optimization with a learning rate decay on each of the optimization instances to obtain a plurality of posture results, the following steps are also included:

[0034] At least one of the illumination model parameters, the reflection parameters, and the occlusion processing parameters is input to configure the differentiable renderer.

[0035] To achieve the above purpose, another aspect of the embodiment of the present application proposes a six-degree-of-freedom object pose estimation system based on differential rendering, the system comprising:

[0036] The first module is used to perform a coarse registration operation of a six-degree-of-freedom pose estimation on a three-dimensional model of an object to obtain a first pose of the object;

[0037] A second module is used to create a number of optimization instances with random initial learning rates according to the first posture;

[0038] The third module is used to use a gradient descent algorithm based on a differentiable renderer and a multimodal loss function to perform posture optimization with a learning rate decay on each of the optimization instances to obtain a plurality of posture results;

[0039] The fourth module is used to take the posture result with the lowest value of the multimodal loss function as the final posture estimation result.

[0040] It should be noted that, in some embodiments, the system may also include the following modules:

[0041] The fifth module is used to input at least one of the illumination model parameters, reflection parameters, and occlusion processing parameters to configure the differentiable renderer.

[0042] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0043] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0044] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for estimating the posture of a 6-DOF object based on differential rendering. The scheme realizes 6-DoF object posture estimation without training data by adopting a differentiable rendering method. It can provide high-precision posture estimation results and has better generalization ability and real-time performance. The transparent optimization process of the scheme enhances the interpretability of the algorithm, and the design of learning rate randomization and parallel optimization instances effectively avoids the local minimum problem and improves the stability and reliability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0046] Figure 1 It is a step diagram of a six-degree-of-freedom object posture estimation method based on differential rendering provided in an embodiment of the present application;

[0047] Figure 2 It is an overall flow chart of a six-degree-of-freedom object posture estimation method based on differential rendering in an embodiment provided in the present application;

[0048] Figure 3is a schematic diagram of a detection frame provided in an embodiment of the present application;

[0049] Figure 4 It is a module schematic diagram of a six-degree-of-freedom object posture estimation system based on differential rendering provided in an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0052] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0053] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0054] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0055] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] In traditional techniques, the Jacobian matrix between the image control and the pose control is used to calculate the incremental pose required to minimize the image-based error. Among them, the neural network compares the rendered image of the 3D mesh with the target image to predict that the rendered image better matches the target model, and then updates the pose. Because these techniques rely on traditional non-rendering pipelines (for example, the 6DoF matrix obtained by direct regression is not necessarily reversible), the neural network must be trained offline on large datasets to learn to calculate pose updates. Once trained, the network is opaque and it is difficult to improve performance without expensive and time-consuming retraining because it is impossible to understand why it performs well or fails.

[0058] In view of this, a six-degree-of-freedom object posture estimation method and system based on differential rendering is provided in an embodiment of the present application. The scheme uses the 6-DoF object posture optimization problem as a direct end-to-end optimization to perform posture estimation. By adopting a differentiable rendering method, 6-DoF object posture estimation without training data is achieved. It can provide high-precision posture estimation results while having better generalization ability and real-time performance. The transparent optimization process of the scheme enhances the interpretability of the algorithm, while the design of learning rate randomization and parallel optimization instances effectively avoids the local minimum problem and improves the stability and reliability of the algorithm. In addition, the flexibility and adaptability of the embodiments of the present application allow it to fuse multiple image information according to specific application scenarios, thereby simplifying the deployment process and reducing costs, and providing strong support for technological progress and application innovation in the fields of robot vision, augmented reality, and autonomous driving navigation.

[0059] A six-degree-of-freedom object posture estimation method based on differential rendering provided in an embodiment of the present application relates to the field of computer technology. A six-degree-of-freedom object posture estimation method based on differential rendering provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a six-degree-of-freedom object posture estimation method based on differential rendering, etc., but is not limited to the above forms.

[0060] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0061] Figure 1 is an optional flow chart of a six-degree-of-freedom object posture estimation method based on differential rendering provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S400.

[0062] Step S100, performing a coarse registration operation of six-degree-of-freedom pose estimation on the three-dimensional model of the object to obtain a first pose of the object.

[0063] Step S200: creating a plurality of optimization instances with random initial learning rates according to the first posture.

[0064] Step S300, using a gradient descent algorithm based on a differentiable renderer and a multimodal loss function, performs posture optimization with a learning rate decay on each of the optimization instances to obtain a plurality of posture results.

[0065] Step S400: taking the posture result with the lowest value of the multimodal loss function as the final posture estimation result.

[0066] Steps S100 to S400 shown in the embodiment of the present application realize 6-DoF object pose estimation without training data by adopting differentiable rendering, which can provide high-precision pose estimation results while having better generalization ability and real-time performance. The transparent optimization process of this solution enhances the interpretability of the algorithm, while the design of learning rate randomization and parallel optimization instances effectively avoids the local minimum problem and improves the stability and reliability of the algorithm.

[0067] In some embodiments, step S100 may include but is not limited to one of steps S110 to S130:

[0068] Step S110, performing six-degree-of-freedom pose estimation on the three-dimensional model of the object through a feature point matching algorithm to obtain a first pose of the object.

[0069] Step S120 , alternatively, by using a template matching algorithm, six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object to obtain a first pose of the object.

[0070] Step S130, alternatively, using a deep learning model, six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object to obtain a first pose of the object.

[0071] In some embodiments, step S200 may include but is not limited to the following steps S210 to S230:

[0072] Step S210, configuring the upper bound and the lower bound of the learning rate.

[0073] Step S220: randomly determine a learning rate between the upper bound of the learning rate and the lower bound of the learning rate as an initial learning rate.

[0074] Step S230: creating an optimization instance according to the first posture and the initial learning rate.

[0075] In some embodiments, step S300 may include but is not limited to the following steps S310 to S350:

[0076] Step S310: acquiring an observation image of the three-dimensional model of the object.

[0077] Step S320: Rendering a two-dimensional image of the three-dimensional model of the object according to the first posture, the three-dimensional model of the object and the camera intrinsic parameters through a differentiable renderer to obtain a rendered image.

[0078] Step S330, calculating the loss value and the loss function gradient between the rendered image and the observed image according to the multimodal loss function; wherein the loss function gradient is the gradient of the multimodal loss function relative to the first posture.

[0079] Step S340: performing posture update according to the first posture, the current learning rate and the loss function gradient to obtain a second posture.

[0080] Step S350, according to the learning rate attenuation factor, update the current learning rate, and use the second posture as the first posture, repeatedly execute the step of rendering the two-dimensional image of the three-dimensional model of the object according to the first posture, the three-dimensional model of the object and the camera intrinsic parameters through the differentiable renderer to obtain the rendered image, until the number of iterations reaches a preset iteration number threshold, and obtains the posture result.

[0081] In step S300 of some embodiments, the expression of the multimodal loss function is:

[0082] L=λ c L x +λ d L d +λ e L e ;

[0083]

[0084]

[0085] Among them, L represents the multimodal loss function; L c is the RGB modality loss; L d is the depth modality loss; L e is the marginal mode loss; and λ c is the weight of the RGB mode; d is the weight of the depth modality; λ e is the weight of the edge mode; L EIoU represents EIoU loss; ⊙ represents element-by-element product; f function represents the minimum detection box of the image; I c is the observed RGB image; is to render RGB images; I d is the observed depth image; is the rendered depth image; I e is the observed edge map; is the rendered edge map; S represents the segmentation mask.

[0086] In step S340 of some embodiments, the expression for performing posture update is:

[0087]

[0088] in, The second posture. is the first posture, α i is the current learning rate, is the gradient of the loss function.

[0089] Before step S100 in some embodiments, a step of inputting at least one of illumination model parameters, reflection parameters, and occlusion processing parameters to configure the differentiable renderer may also be included.

[0090] Below, in conjunction with a specific application example of six-degree-of-freedom object posture estimation, the solution of the embodiment of the present application is described in detail and explained:

[0091] Coarse registration and fine registration in 6-DoF pose estimation are two consecutive steps in the object pose estimation process, and their common goal is to accurately determine the position and orientation of the object in three-dimensional space.

[0092] The goal of coarse registration is to quickly estimate a rough object pose, which usually involves using the object's geometry or feature information to determine a preliminary pose, usually done through feature point matching, template matching, or using deep learning models that can recognize objects and provide a preliminary pose estimate.

[0093] Fine registration aims to further improve the accuracy of pose estimation through a more sophisticated optimization process based on the pose provided by coarse registration. The embodiment of the present application provides a six-degree-of-freedom object pose estimation method based on differential rendering, which focuses on fine registration after coarse registration of 6-DoF pose estimation to achieve higher-precision six-degree-of-freedom object pose estimation without the need for training data, and has high robustness and flexibility, and can select and fuse multiple image information, such as RGB, depth map, edge detection and semantic segmentation.

[0094] During implementation, the embodiments of the present application may include three parts: differentiable renderer setting, loss function construction and gradient calculation, posture optimization and optimal posture output.

[0095] (1) Differentiable renderer settings.

[0096] The goal of the embodiment of the present application to achieve high-precision pose estimation is to find a transformation T from the camera coordinate system to the object coordinate system given a 3D model M and a camera intrinsic parameter K. CO , so that the rendered image The image that is closest to the observed image I.

[0097] Furthermore, the embodiment of the present application configures a differentiable renderer to render the reprojection of the 3D model under the intrinsic parameter k posture given by the camera, and then performs 6-DoF pose estimation by minimizing the reprojection error to calculate T CO The expression is:

[0098]

[0099] Among them, argminloss() is loss minimization, T CO Finally, from all possible transformations T CO ′ selects the transformation that minimizes the loss function as the output; R is a differentiable renderer that accepts the object pose T CO , 3D model M and camera intrinsic parameter K as input, output rendered image

[0100] The input parameters received by the differentiable renderer are continuously differentiable, so that the rendered image can be calculated through the back-propagation algorithm. Relative to the object pose T CO Gradient in, Represents partial derivative.

[0101] Based on this, the embodiment of the present application first configures a differentiable renderer, sets other parameters such as illumination model, reflection, occlusion processing, etc. according to needs, and uses the differentiable renderer to generate a differentiable image according to the first posture. Render the 2D image of the 3D model M to obtain a rendered image. The expression of the rendered image is:

[0102]

[0103] At this point, the differentiable renderer is set up and uses the initialization settings and the 3D model to render a 2D image, which is used in the next step to calculate the difference between the rendered image and the observed image, ultimately achieving accurate fine registration.

[0104] (2) Loss function construction and gradient calculation.

[0105] Although the overall loss can be obtained through a single RGB image, in the real world, due to problems such as lighting and texture reflection on the surface of objects, using only one modality can easily cause the model to be less robust. Therefore, the loss function L integrates information from different image modalities (such as RGB, depth, and edges) (other modalities can also be added based on the data obtained by the camera, such as point clouds, etc.) to accurately measure the difference between the rendered image and the observed image.

[0106] The loss function is calculated based on the weighted sum of the losses of each modality. The expression of the loss function is:

[0107] L=λ c L c +λ d L d +λ e L e ;

[0108] Among them, L c is the RGB modality loss, L d is the depth modality loss, L e is the marginal modal loss, and λ c , d , e are the weights of the corresponding modes respectively.

[0109] For each modality m, the loss L m The calculation is as follows (the loss definitions of each modality are basically the same). The instance segmentation model SAM is used to obtain the mask of the object and calculate the minimum detection box of the target, which is recorded as [x, y, w, h], which represents the horizontal coordinate of the center point, the vertical coordinate of the center point, the width, and the height, respectively. Figure 3 As shown. EIoU is selected as the training loss function. The EIoU loss function is a continuous and differentiable function. It is an indicator used to measure the degree of overlap between the predicted box and the real box in the target detection task. EIoU adds two parts based on the IOU calculation, distance loss and height-width loss (overlapping area, center point example, aspect ratio). The height-width loss directly minimizes the difference in height and width between the predicted target bounding box and the real bounding box, so that it has a faster convergence speed and better positioning results. Therefore, the EIoU loss can be calculated as:

[0110]

[0111] Among them, w c and h c is the width and height of the minimum bounding rectangle of the predicted target bounding box and the true bounding box. b, w, h are the center point, width and height of the predicted target bounding box respectively; b gt 、w gt 、hgt are the center point, width and height of the ground-truth bounding box respectively; ρ is the Euclidean distance between two points.

[0112] IoU is calculated by dividing the intersection of the predicted box (A) and the true box (B) by the union of the two. The formula is:

[0113]

[0114] Among them, w c and h c is the width and height of the minimum bounding rectangle of the predicted target bounding box and the true bounding box. w and h are the width and height of the predicted target bounding box, respectively. gt 、h gt are the width and height of the ground-truth bounding box, respectively.

[0115] The improved EIoU splits the aspect ratio loss into the difference between the predicted width and height and the minimum bounding box width and height, which accelerates convergence and improves regression accuracy.

[0116] Based on this, the modal loss of the embodiment of the present application is calculated as follows:

[0117] (1) RGB modality loss L c :

[0118]

[0119] Among them, I c is the observed RGB image, is the rendered RGB image, S is the object segmentation mask, ⊙ represents the element-by-element product, and the f function represents the minimum detection box of the image. The modal Loss is calculated by calculating the difference between the object detection boxes in the rendered image and the original image.

[0120] (2) Deep modal loss L d :

[0121]

[0122] Among them, I d is the observation depth map, is the rendered depth map.

[0123] (3) Marginal modal loss L e :

[0124]

[0125] Among them, I e is the observed edge map, is the rendered edge map.

[0126] The segmentation mask S is used to limit the calculation of the loss function to the object area, ignoring the background and interference:

[0127] S = {0 (background), 1 object)};

[0128] You can use tools such as Sam, which is based on a large zero-shot image segmentation model, to obtain high-quality segmentation masks. Sam is a large zero-shot image segmentation model launched by Meta AI. It can quickly complete image segmentation and obtain high-quality object masks by interactively or automatically segmenting given prompt words, such as the text "cat and chair".

[0129] Furthermore, in order to optimize the object posture, it is necessary to calculate the gradient of the loss function relative to the object posture. Specifically, it is necessary to calculate the gradient of the loss function of each modality:

[0130]

[0131] Among them, T CO Refers to the transformation from the camera coordinate system to the object coordinate system, L m Refers to the loss of the above single mode.

[0132] According to the chain rule, the overall loss function is the weighted sum of multiple modal losses:

[0133]

[0134] Through the above steps, all necessary information is prepared, a loss function of multimodal information is constructed, and the gradient for posture optimization is calculated.

[0135] (3) Posture optimization and optimal posture output.

[0136] The specific steps of the embodiment of the present application can be designed as follows Figure 2 As shown, before starting the optimization, the relevant parameters are initialized, including the number of iterations iters, the batch size B, and the upper bound of the learning rate and upper bound h.

[0137] A coarse registration operation of six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object to obtain a first pose of the object; based on the first pose, several optimization instances with random initial learning rates are created.

[0138] The gradient descent algorithm based on the differentiable renderer and the multimodal loss function are used to perform posture optimization with learning rate decay for each optimized instance. Specifically, for each optimized instance i: Randomly select a learning rate α between i , from the initial posture To begin, use the learning rate α iMake optimizations.

[0139] At each iteration k, the object pose is updated using gradient descent:

[0140]

[0141] in, It is the second posture, the new posture; is the first posture, the old posture; α i is the current learning rate; is the loss function L with respect to the pose T CO gradient.

[0142] The current learning rate is decayed according to the learning rate decay factor to prevent oscillation near the optimal solution. The decay calculation of the current learning rate is:

[0143]

[0144] in, is the current learning rate at the kth iteration.

[0145] After each iteration, the loss function L corresponding to the current posture is evaluated. After all optimization instances are completed, the posture with the lowest loss function value is selected as the final posture estimation. The expression of the final posture estimation result is:

[0146]

[0147] in, is the final pose estimation result; B is the number of optimized instances, that is, the batch size.

[0148] In summary, the embodiments of the present application have at least the following beneficial effects: a 6-DoF object posture estimation method without training data is implemented, which can provide high-precision posture estimation results, while having better generalization ability and real-time performance. The transparent optimization process of the embodiments of the present application enhances the interpretability of the algorithm, and the design of learning rate randomization and parallel optimization instances effectively avoids the local minimum problem and improves the stability and reliability of the algorithm. In addition, the flexibility and adaptability of the embodiments of the present application allow it to fuse multiple image information according to specific application scenarios, thereby simplifying the deployment process and reducing costs, and providing strong support for technological progress and application innovation in the fields of robot vision, augmented reality, and autonomous driving navigation.

[0149] See also Figure 4 The embodiment of the present application further provides a six-degree-of-freedom object posture estimation system based on differential rendering, which can implement the above-mentioned six-degree-of-freedom object posture estimation method based on differential rendering, and the system includes:

[0150] The first module 101 is used to perform a coarse registration operation of a six-degree-of-freedom pose estimation on a three-dimensional model of an object to obtain a first pose of the object;

[0151] A second module 102 is used to create a plurality of optimization instances with random initial learning rates according to the first posture;

[0152] The third module 103 is used to use a gradient descent algorithm based on a differentiable renderer and a multimodal loss function to perform posture optimization with a learning rate decay on each of the optimization instances to obtain a plurality of posture results;

[0153] The fourth module 104 is used to take the posture result with the lowest value of the multimodal loss function as the final posture estimation result.

[0154] It should be noted that, in some embodiments, the system may also include the following modules:

[0155] The fifth module is used to input at least one of the illumination model parameters, reflection parameters, and occlusion processing parameters to configure the differentiable renderer.

[0156] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0157] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned six-degree-of-freedom object posture estimation method based on differential rendering when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0158] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0160] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0161] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes a six-degree-of-freedom object posture estimation method based on differential rendering in the embodiment of the present application;

[0162] Input / output interface 203, used to implement information input and output;

[0163] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0164] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0165] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0166] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned six-degree-of-freedom object posture estimation method based on differential rendering.

[0167] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0168] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0170] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0171] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0172] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0173] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0174] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0176] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0179] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A six-degree-of-freedom object posture estimation method based on differential rendering, characterized in that: The following steps are involved: Performing a coarse registration operation of six-degree-of-freedom pose estimation on the three-dimensional model of the object to obtain a first pose of the object; According to the first posture, create several optimization instances with random initial learning rates; Using a gradient descent algorithm based on a differentiable renderer and a multimodal loss function, performing posture optimization with a learning rate decay on each of the optimization instances, and obtaining a number of posture results; The posture result with the lowest value of the multimodal loss function is taken as the final posture estimation result.

2. The method according to claim 1, characterized in that The coarse registration operation of the three-dimensional model of the object with six degrees of freedom pose estimation is performed to obtain a first pose of the object, Include one of the following steps: By using a feature point matching algorithm, a six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object to obtain a first pose of the object; Alternatively, a six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object through a template matching algorithm to obtain a first pose of the object; Alternatively, a six-degree-of-freedom pose estimation is performed on the three-dimensional model of the object through a deep learning model to obtain a first pose of the object.

3. The method according to claim 1, characterized in that In the step of creating a plurality of optimization instances with random initial learning rates according to the first posture, the step of creating the optimization instances comprises the following steps: Configure the upper and lower bounds of the learning rate; Randomly determine a learning rate between the learning rate upper bound and the learning rate lower bound as the initial learning rate; An optimization instance is created according to the first posture and the initial learning rate.

4. The method according to claim 1, characterized in that: In the step of adopting a gradient descent algorithm based on a differentiable renderer and a multimodal loss function to perform posture optimization with a learning rate decay on each of the optimization instances to obtain a plurality of posture results, performing posture optimization on the optimization instance includes the following steps: Acquiring an observation image of the three-dimensional model of the object; Rendering a two-dimensional image of the three-dimensional model of the object according to the first posture, the three-dimensional model of the object and the camera intrinsic parameters by a differentiable renderer to obtain a rendered image; According to the multimodal loss function, calculating the loss value and the loss function gradient between the rendered image and the observed image; wherein the loss function gradient is the gradient of the multimodal loss function relative to the first posture; Performing posture updating according to the first posture, the current learning rate and the gradient of the loss function to obtain a second posture; According to the learning rate attenuation factor, the current learning rate is updated, and the second posture is used as the first posture, and the step of rendering a two-dimensional image of the three-dimensional model of the object according to the first posture, the three-dimensional model of the object and the camera intrinsic parameters through a differentiable renderer to obtain a rendered image is repeatedly performed until the number of iterations reaches a preset iteration number threshold, and a posture result is obtained.

5. The method according to claim 1, characterized in that The expression of the multimodal loss function is: L=λ c L c +λ d L d +λ e L e ; Where L represents the multimodal loss function; L c is the RGB modality loss; L d is the depth modality loss; L e is the marginal mode loss; and λ c is the weight of the RGB mode; d is the weight of the depth modality; λ e is the weight of the edge mode; L EIoU represents EIoU loss; ⊙ represents element-by-element product; f function represents the minimum detection box of the image; I c is the observed RGB image; is to render RGB images; I d is the observed depth image; is the rendered depth image; I e is the observed edge map; is the rendered edge map; S represents the segmentation mask.

6. The method according to claim 4, characterized in that The expression for posture update is: in, The second posture. is the first posture, α i is the current learning rate, is the gradient of the loss function.

7. The method according to claim 1, characterized in that Before adopting the gradient descent algorithm based on the differentiable renderer and the multimodal loss function to perform posture optimization with a learning rate decay on each of the optimization instances to obtain a plurality of posture results, the following steps are also included: At least one of the illumination model parameters, the reflection parameters, and the occlusion processing parameters is input to configure the differentiable renderer.

8. A six-degree-of-freedom object posture estimation system based on differential rendering, characterized in that: include: The first module is used to perform a coarse registration operation of a six-degree-of-freedom pose estimation on a three-dimensional model of an object to obtain a first pose of the object; A second module is used to create a number of optimization instances with random initial learning rates according to the first posture; The third module is used to use a gradient descent algorithm based on a differentiable renderer and a multimodal loss function to perform posture optimization with a learning rate decay on each of the optimization instances to obtain a plurality of posture results; The fourth module is used to take the posture result with the lowest value of the multimodal loss function as the final posture estimation result.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images