Position and orientation acquisition method, device, electronic device, storage medium, and program

By acquiring sampling points on a search line segment and constructing an objective function based on attribute information and weights, the method improves the accuracy of position and orientation parameters, addressing interference issues in augmented reality systems.

JP7595159B2Active Publication Date: 2024-12-05ZHEJIANG SENSETIME TECH DEV CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023525620
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-06-04
Filing Date
2021-10-29
Publication Date
2024-12-05
Estimated Expiration
2041-10-29

AI Technical Summary

Technical Problem

The accuracy of position and orientation parameters in real scenes is often compromised by interference factors such as local occlusion and similar colors, leading to deviations in virtual and real object matching in augmented reality systems.

Method used

A method involving the acquisition of sampling points on a search line segment passing through a projected contour point, where attribute information and reference weights are used to construct an objective function, improving the accuracy of position and orientation parameters by filtering interference factors and optimizing pose parameters.

Benefits of technology

Enhances the accuracy of position and orientation estimation by reducing the influence of interference factors, enabling precise virtual-real fusion with geometric consistency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595159000266
    Figure 0007595159000266
  • Figure 0007595159000267
    Figure 0007595159000267
  • Figure 0007595159000268
    Figure 0007595159000268
Patent Text Reader

Abstract

The present application discloses a position and orientation acquisition method, an apparatus therefor, an electronic device, a storage medium, and a program therefor, wherein the position and orientation acquisition method includes acquiring several sampling points located on a search line segment in a captured image, where the search line segment passes through projected contour points of a target object in the captured image and the projected contour points are located on the projected contour of the target object; acquiring attribute information of the sampling points and reference weights of the sampling points, where the attribute information represents a possibility that the sampling points belong to the target object; constructing an objective function based on the attribute information and reference weights of the sampling points; and acquiring position and orientation parameters of the target object in the captured image based on the objective function.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is filed on June 06, 2021. 4 This application claims priority to a Chinese patent application filed on the same day with the China Patent Office, bearing application number 202110626207.X, filed on the same day by Zhejiang SenseTime Technology Development Company, Limited and Shandong University, and entitled "Position and attitude acquisition method and apparatus, as well as electronic device and storage medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer vision technology, and relates to, but is not limited to, a position and orientation acquisition method and an apparatus therefor, an electronic device, a storage medium, and a program. [Background technology]

[0003] With the development of information technology, position and orientation parameters are increasingly widely applied in many scenarios, such as augmented reality systems, robot hand-eye calibration, interactive games, human-machine interaction, etc. For example, in an augmented reality system, a virtual object can be rendered according to the position and orientation parameters and superimposed on a real object in a video image, thereby realizing a virtual-real fusion effect with spatial and geometric consistency.

[0004] Currently, in real scenes, the accuracy of position and pose parameters is often seriously affected by interference factors such as local occlusion, similar colors, etc. In view of the above problems, how to improve the accuracy of position and pose parameters has become a problem to be solved. Summary of the Invention

[0005] The present embodiment provides a position and orientation acquisition method, an apparatus therefor, an electronic device, a storage medium, and a program.

[0006] In a first aspect of the present embodiment, there is provided a position and orientation acquisition method applied to an electronic device, the method comprising: The method includes obtaining several sampling points located on a search line segment in the captured image, where the search line segment passes through projected contour points of a target object in the captured image, and the projected contour points are located on the projected contour of the target object; obtaining attribute information of the sampling points and obtaining reference weights of the sampling points, where the attribute information represents a possibility that the sampling points belong to the target object; constructing an objective function based on the attribute information and the reference weights of the sampling points; and obtaining position and orientation parameters of the target object in the captured image based on the objective function.

[0007] Therefore, some sampling points located on a search line segment in the captured image are obtained, the search line segment passes through the projected contour points of the target object in the captured image, and the projected contour points are located on the projected contour of the target object, and based on this, attribute information of the sampling points is then obtained, and reference weights of the sampling points are obtained, the attribute information representing the possibility that the sampling points belong to the target object, so that an objective function is constructed based on the attribute information and reference weights of the sampling points, and the position and orientation parameters of the target object in the captured image are obtained based on the objective function. The objective function is constructed based on both the attribute information and reference weights of the sampling points, where the attribute information can reflect the possibility that the sampling points belong to the target object, and the reference weights can reflect the reference value of the sampling points in the subsequent process of solving the position and orientation parameters, so that when the objective function is constructed based on the attribute information and reference weights of the sampling points, the influence of interference factors on the position and orientation solving can be reduced and the accuracy of the position and orientation parameters can be improved.

[0008] In one embodiment, obtaining reference weights for the sampling points includes: searching for a target point from several sampling points on a search line segment to obtain a search result, where the target point is used to represent an object contour point of the target object; obtaining weight information for several sampling points on the search line segment based on the search result, respectively, where the weight information includes at least one of a first weight and a second weight, where the first weight is correlated with a predicted probability value of the target point, the predicted probability value representing a possibility that the sampling point is an object contour point, and the second weight is correlated with a first distance from the target point to the sampling point; and obtaining reference weights for the sampling points based on the weight information.

[0009] Therefore, a target point is searched for from several sampling points on the search line segment to obtain a search result, and the target point is used to represent an object contour point of the target object. Based on the search result, weight information of several sampling points on the search line segment is obtained respectively, and the weight information includes at least one of a first weight and a second weight, where the first weight is correlated with a predicted probability value of the target point, and the predicted probability value represents the possibility that the sampling point is an object contour point, and the second weight is correlated with a first distance from the target point to the sampling point, so that the first weight and the second weight can represent the reference value of the sampling point from different angles. Based on this, then, based on the weight information, the reference weight of the sampling point is obtained, so that the reference value of the reference weight in the subsequent process of solving the position and posture parameters can be improved.

[0010] In one embodiment, the attribute information includes a first probability value that the sampling point belongs to the target object, and searching for a target point from several sampling points on the search line segment to obtain a search result includes: for each search line segment, using several sampling points as current points respectively, and using the current point as a candidate point if a reference probability difference of the current point satisfies a first condition, and selecting a candidate point whose predicted cost value satisfies a second condition as a target point, where the reference probability difference of the current point is a difference between first probability values ​​of two of the sampling points having a predetermined positional relationship with the current point, and the predicted cost value includes at least one of a first cost value and a second cost value, the first cost value being correlated with a predicted probability value of the candidate point, and the second cost value being correlated with a second distance from the candidate point to a projected contour point on the search line segment.

[0011] Therefore, the attribute information includes a first probability value that the sampling point belongs to the target object, for each search line segment, several sampling points are respectively used as current points, if the reference probability difference of the current point satisfies a first condition, the current point is used as a candidate point, and the candidate point whose predicted cost value satisfies a second condition is selected as a target point, the reference probability difference of the current point is a difference between the first probability values ​​of two of the sampling points having a predetermined positional relationship with the current point, the predicted cost value includes at least one of a first cost value and a second cost value, the first cost value is correlated with the predicted probability value of the candidate point, and the second cost value is correlated with a second distance from the candidate point to a projected contour point on the search line segment, that is, the first cost value and the second cost value respectively represent the costs of the candidate point regarded as an object contour point from different angles, therefore, the efficiency and accuracy of the selection of the target point can be improved by first obtaining the candidate point by a rough selection based on the reference probability difference, and then obtaining the target point by a precise selection based on the predicted cost value.

[0012] In one embodiment, before selecting the candidate point whose predicted cost value satisfies the second condition as the target point, the method further includes filtering candidate points whose predicted probability value satisfies a third condition.

[0013] Therefore, before accurately selecting a target point from the candidate points, the candidate points whose predicted probability values ​​satisfy the third condition are first filtered out, and the predicted probability value represents the possibility that the sampling point is an object contour point, so that the selection efficiency of the target point can be further improved.

[0014] In one embodiment, the predetermined positional relationship is adjacent to the current point, and / or the second condition includes the predicted cost value being the smallest, and / or the first cost value is negatively correlated with the predicted probability value of the candidate point and the second cost value is positively correlated with the second distance.

[0015] Therefore, by setting the specified positional relationship to be adjacent to the current point, it is possible to accurately evaluate the sudden change in the first probability value of each sampling point and improve the accuracy of the candidate point; by setting the second condition to include the smallest predicted cost value, it is possible to reduce the influence of interference factors on the selection of the target point as much as possible and improve the accuracy of the position and orientation parameters; and by setting the first cost value to be negatively correlated with the predicted probability value of the candidate point and the second cost value to be positively correlated with the second distance, it is possible to improve the accuracy of the first cost value and the second cost value.

[0016] In one embodiment, the weight information includes a first weight, and obtaining weight information for each of several sampling points on the search line segment based on the search result includes: determining a first weight for the sampling point based on a predicted probability value of the target point when the search result includes that the target point is searched, where the first weight is positively correlated with the predicted probability value of the target point; and / or determining the first weight as a first numerical value when the search result includes that the target point is not searched, where the first numerical value is a lower limit value of the first weight when the search result includes that the target point is searched.

[0017] Therefore, the weight information includes a first weight, and when the search result includes that the target point is searched, a first weight of the sampling point is determined based on the predicted probability value of the target point, the first weight is positively correlated with the predicted probability value of the target point, and when the search result includes that the target point is not searched, the first weight is determined as a first numerical value, and the first numerical value is a lower limit value of the first weight when the search result includes that the target point is searched, so that the entire search line segment can be used as a dimension to determine the first weight of each sampling point on the search line segment, and the efficiency of obtaining the first weight can be improved.

[0018] In one embodiment, the weight information includes a second weight, and obtaining weight information for each of several sampling points on the search line segment based on the search result includes: determining a second weight for the sampling point based on a first distance corresponding to the sampling point when the search result includes that the target point is searched, where the second weight is negatively correlated with the first distance; and / or determining the second weight as a second numerical value when the search result includes that the target point is not searched, where the second numerical value is an upper limit value of the second weight when the search result includes that the target point is not searched.

[0019] Therefore, the weight information includes a second weight, and when the search result includes that the target point is searched, a second weight of the sampling point is determined based on the first distance corresponding to the sampling point, the second weight is negatively correlated with the first distance, and when the search result includes that the target point is not searched, the second weight is determined as a second numerical value, and the second numerical value is an upper limit value of the second weight when the search result includes that the target point is not searched, so that the entire search line segment can be used as a dimension to determine the second weight of each sampling point on the search line segment, and the efficiency of obtaining the second weight can be improved.

[0020] In one embodiment, the weight information includes a first weight and a second weight, both of which are positively correlated with the reference weight.

[0021] Therefore, by setting the weight information to include a first weight and a second weight, and both the first weight and the second weight to be positively correlated with the reference weight, the reference value of the sampling point in the subsequent process of solving the position and orientation parameters can be expressed from two different dimensions of the first weight and the second weight, and the reference value of the reference weight itself can be improved.

[0022] In one embodiment, the attribute information includes a first probability value and a first reliability that the sampling point belongs to the target object, and a second probability value and a second reliability that the sampling point does not belong to the target object, and constructing the objective function based on the attribute information of the sampling point and the reference weights includes obtaining a first product of the first reliability and the first probability value and a second product of the second reliability and the second probability value, obtaining a joint probability of the sampling point based on a sum of the first product and the second product, and obtaining the objective function based on a weighting result of the joint probability by the reference weight of each sampling point.

[0023] Therefore, the attribute information includes a first probability value and a first reliability that the sampling point belongs to the target object, and a second probability value and a second reliability that the sampling point does not belong to the target object. Based on this, a first product of the first reliability and the first probability value, and a second product of the second reliability and the second probability value are obtained, and the joint probability of the sampling point is obtained based on the sum of the first product and the second product. Thus, the joint probability of the sampling point can be expressed from the angle at which the sampling point belongs to the target object and the angle at which the sampling point does not belong to the target object. By constructing an objective function based on the result of weighting the joint probability by the reference weight of each sampling point, the accuracy of the objective function can be improved, and the accuracy of the reference position and orientation can be improved.

[0024] In one embodiment, the first reliability is negatively correlated with the second reliability, the first reliability of a sampling point is negatively correlated with a directed Euclidean distance from a corresponding projected contour point to the sampling point, and the corresponding projected contour point and the sampling point are located on the same search line segment.

[0025] Therefore, the first reliability has a negative correlation with the second reliability, and the first reliability of a sampling point has a negative correlation with the oriented Euclidean distance from the corresponding projected contour point to the sampling point, and the corresponding projected contour point and sampling point are located on the same search line segment; that is, the smaller the oriented Euclidean distance, the higher the first reliability that the sampling point belongs to the target object, and the lower the second reliability that the sampling point does not belong to the target object, so that the influence of interference factors such as local occlusion can be reduced as much as possible.

[0026] In one embodiment, the captured image includes a foreground region and a background region divided based on a projected contour, and before obtaining a first product of the first reliability and the first probability value and a second product of the second reliability and the second probability value, the position and orientation acquisition method further includes filtering the sampling point if the oriented Euclidean distance of the sampling point is greater than the first distance value and the sampling point belongs to the foreground region, and / or filtering the sampling point if the oriented Euclidean distance of the sampling point is less than the second distance value and the sampling point belongs to the background region.

[0027] Therefore, the captured image includes a foreground region and a background region divided based on the projected contour, and before calculating the joint probability, first, a sampling point having a directional Euclidean distance greater than the first distance value and belonging to the foreground region is detected, and then the sampling point is regarded as an interference point and filtered, thereby reducing the influence of the interference point on solving the subsequent position and posture parameters as much as possible; first, a sampling point having a directional Euclidean distance smaller than the second distance value and belonging to the background region is detected, and then the sampling point is regarded as an interference point and filtered, thereby reducing the influence of the interference point on solving the subsequent position and posture parameters as much as possible.

[0028] In one embodiment, the projected contour is obtained by projecting using a reference position and orientation of the target object, and before obtaining several sampling points located on the search line segment in the captured image, the position and orientation acquisition method includes: down-sampling the captured image to obtain pyramid images of several resolutions; sequentially selecting the pyramid images as the current captured image according to the resolution from small to large; and performing the steps of obtaining several sampling points located on the search line segment in the captured image for the current captured image, wherein the reference position and orientation adopted in the current run are the position and orientation parameters obtained in the previous run, and the position and orientation parameters obtained in the last run are used as the final position and orientation parameters of the target object in the captured image.

[0029] Therefore, the projected contour is obtained by projecting using the reference position and orientation of the target object, so that before the projection sampling, first, downsample the captured image to obtain pyramid images of several resolutions, sequentially select the pyramid images as the current captured image according to the resolution from small to large, and perform the steps of obtaining several sampling points located on the search line segment in the above captured image for the current captured image, and the reference position and orientation adopted in this execution are the position and orientation parameters obtained in the previous execution, and the position and orientation parameters obtained in the last execution are used as the final position and orientation parameters of the target object in the captured image, so that in the process of obtaining the position and orientation parameters, position and orientation estimation can be performed in the order from rough position and orientation estimation to accurate position and orientation estimation, and the efficiency and accuracy of obtaining the position and orientation parameters can be further improved.

[0030] In one embodiment, the projected contour is obtained by projecting using a reference pose of the target object, and obtaining pose parameters of the target object in the captured image based on the objective function includes solving the objective function to obtain update parameters of the reference pose, and optimizing the reference pose using the update parameters to obtain the pose parameters.

[0031] Therefore, the projected contour is obtained by projecting using a reference pose of the target object, where the reference pose is the pose parameters of the target object in a reference image, which is captured before the captured image. The objective function is solved to obtain update parameters of the reference pose, and the update parameters are optimized to obtain the pose parameters, thereby enabling the pose parameters to be accurately and continuously tracked during the capture process of the target object.

[0032] In a second aspect of the present embodiment, there is provided a position and orientation acquisition device to be applied to an electronic device, the device comprising: The present invention comprises a projection sampling module, an information extraction module, a function construction module, and a position and orientation solving module, wherein the projection sampling module is configured to obtain a number of sampling points located on a search line segment in a captured image, where the search line segment passes through a projected contour point of a target object in the captured image, and the projected contour point is located on the projected contour of the target object, the information extraction module is configured to obtain attribute information of the sampling points and obtain reference weights of the sampling points, where the attribute information represents a possibility that the sampling points belong to the target object, the function construction module is configured to construct an objective function based on the attribute information and the reference weights of the sampling points, and the position and orientation solving module is configured to obtain position and orientation parameters of the target object in the captured image based on the objective function.

[0033] In a third aspect of the present embodiment, there is provided an electronic device comprising a memory and a processor coupled to each other, the processor being configured to execute the position and orientation acquisition method in the first aspect above by executing program instructions stored in the memory.

[0034] In a fourth aspect of the present invention, there is provided a computer-readable storage medium having program instructions stored thereon, the program instructions causing a processor to execute the position and orientation acquisition method in the first aspect above.

[0035] In a fifth aspect of the present embodiment, a computer program is provided that includes computer readable code, which, when executed by an electronic device, causes a processor of the electronic device to perform any of the position and orientation acquisition methods described above.

[0036] In the above technical solution, some sampling points located on a search line segment in the captured image are obtained, the search line segment passes through the projected contour points of the target object in the captured image, and the projected contour points are located on the projected contour of the target object, and based on this, attribute information of the sampling points is then obtained, and reference weights of the sampling points are obtained, the attribute information represents the possibility that the sampling points belong to the target object, and thus, an objective function is constructed based on the attribute information and reference weights of the sampling points, and the position and orientation parameters of the target object in the captured image are obtained based on the objective function. The objective function is constructed based on both the attribute information and reference weights of the sampling points, where the attribute information can reflect the possibility that the sampling points belong to the target object, and the reference weights can reflect the reference value of the sampling points in the subsequent process of solving the position and orientation parameters, so that when the objective function is constructed based on the attribute information and reference weights of the sampling points, the influence of interference factors on the position and orientation solving can be reduced and the accuracy of the position and orientation parameters can be improved.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the invention. Other features and aspects of the embodiments of the invention will become apparent from the following detailed description of illustrative embodiments taken in conjunction with the drawings. [Brief description of the drawings]

[0038] [Figure 1] 1 is a flowchart of an embodiment of a position and orientation acquisition method according to the present application. [Diagram 2] FIG. 2 is a schematic diagram of one embodiment of a contour mask. [Diagram 3] FIG. 2 is a schematic diagram of one embodiment of a projected contour and a search line segment; [Figure 4a] FIG. 2 is a schematic diagram of one example of a captured image. [Figure 4b] FIG. 4 is a schematic diagram of another example of a mask image. [Figure 4c] FIG. 2 is a schematic diagram of one embodiment of a search line segment. [Figure 5a] FIG. 13 is a schematic diagram of another example of a captured image. [Figure 5b] FIG. 2 is a schematic diagram of one embodiment of a layout region. [Figure 5c] FIG. 2 is a schematic diagram of another example of a local region. [Figure 6] 2 is a flowchart of one embodiment of step S12 in FIG. 1. [Figure 7a] FIG. 4 is a diagram showing a cluster image of the search line segment in FIG. 3; [Figure 7b] FIG. 4 is a diagram showing a cluster image of first probability values ​​of each sampling point on the search line segment in FIG. 3. [Figure 8] 11 is a flowchart of another embodiment of a position and orientation acquisition method according to the present application. [Figure 9] 1 is a schematic diagram of the framework of one embodiment of a position acquisition device according to the present application. [Figure 10] FIG. 1 is a schematic diagram of the framework of one embodiment of an electronic device according to the present application. [Figure 11]1 is a schematic diagram of the framework of one embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] The drawings herein are incorporated in and constitute a part of this specification, and illustrate embodiments consistent with the present invention, and are used together with this specification to explain the technical solutions of the present invention.

[0040] The solutions in the embodiments of the present application are described in detail below with reference to the drawings.

[0041] In the following description, specific details are given, such as particular system structures, interfaces, and techniques, to provide a thorough understanding of the present application; however, these details are for the purpose of explaining the present application only and are not intended to limit the present application.

[0042] The terms "system" and "network" are always used interchangeably herein. The term "and / or" in this specification is merely for describing the related relationship of related objects, and represents that three relationships may exist, for example, A and / or B can represent three cases: only A exists, both A and B exist, and only B exists. Furthermore, the symbol " / " in this specification generally represents that the relationship between the associated objects is an "or" relationship. Furthermore, the term "multiple" in this specification represents two or more.

[0043] The position and orientation acquisition method provided by the embodiment of the present application can be applied to an augmented reality system, which needs to accurately estimate the position and orientation of a real object relative to a camera, so that a virtual object can be accurately rendered based on the position and orientation parameters and superimposed on the real object in a video to realize a virtual reality blending effect with spatial and geometric consistency.

[0044] However, in practical applications, in a complex environment, there are local occlusions and similar colors, which tend to cause tracking deviations or tracking failures of real objects, resulting in deviations in matching between virtual objects and real objects.

[0045] In order to solve the above technical problems, the position and orientation acquisition method provided in the present embodiment detects and processes pixel points that are locally occluded or interfered with by similar colors, thereby improving the accuracy of object position and orientation estimation, and achieving accurate matching between virtual and reality and a geometrically consistent virtual and reality fusion effect.

[0046] The position and orientation acquisition method provided by the embodiment of the present invention will be described in detail below.

[0047] Referring to Figure 1, Figure 1 is a flowchart of an embodiment of a position and orientation acquisition method according to the present application. Specifically, the method may include the following steps:

[0048] In step S11, several sampling points located on a search line segment in a captured image are obtained.

[0049] In this embodiment, the search line segment passes through the projected contour points of the target object in the captured image, and the projected contour points are located on the projected contour of the target object.

[0050] In one implementation scenario, the projected contour is obtained by projecting using a reference pose of the target object, the reference pose being the pose parameters of the target object in a reference image, and the reference image being captured before the captured image. For example, in a real scene, video data of the target object can be captured, and the video data may include images of multiple frames, in which, for the image of the t-1th frame, the pose parameters of the target object in the image of the t-1th frame can be obtained by the steps in the embodiment of the present application. When we obtain JPEG0007595159000001.jpg78 and obtain the position and orientation parameters of the target object in the t-th frame of the image, JPEG0007595159000002.jpg78 can be used as the reference position and orientation, and the position and orientation parameters of the target object in the image of the t-th frame can be obtained by the steps in this embodiment. JPEG0007595159000003.jpg75 can be obtained, and this analogy can be used, and will not be repeated here.

[0051] In one implementation scenario, in order to improve the convenience of projection, 3D modeling can be performed on the target object in advance to obtain a 3D model of the target object.

[0052] It should be understood that the three-dimensional model may include some vertices and edges connecting the vertices. The specific process of three-dimensional modeling may be referred to technical details related to three-dimensional modeling, and will not be described in detail here.

[0053] In one implementation, for ease of explanation, the reference pose is: It can be written as JPEG0007595159000004.jpg54 and can be represented as a 4*4 homogeneous matrix as follows. JPEG0007595159000005.jpg1852…(1)

[0054] In the above formula (1), JPEG0007595159000006.jpg816 represents the Special Euclidean Group, JPEG0007595159000007.jpg44 represents the rotation parameters, JPEG0007595159000008.jpg53 represents the translation parameters, JPEG0007595159000009.jpg44 JPEG0007595159000010.jpg715 (i.e., Special Orthogonal Group), JPEG0007595159000011.jpg53 is a real sequence. Based on this, the camera internal parameters JPEG0007595159000012.jpg45 and the above reference position and orientation 3D points on the target object using JPEG0007595159000013.jpg54 JPEG0007595159000014.jpg45 is projected onto the captured image to obtain 3D points Pixel points in the captured image corresponding to JPEG0007595159000015.jpg45 You can get JPEG0007595159000016.jpg34. JPEG0007595159000017.jpg975

[0055] In the above formula (2), JPEG0007595159000018.jpg848, JPEG0007595159000019.jpg54 is a 3D point represents the homogeneous coordinates of JPEG0007595159000020.jpg45, i.e. JPEG0007595159000021.jpg647, and the 3D point The normal coordinates for JPEG0007595159000022.jpg45 are In addition, in the persistent tracking scene described above, the relative position and orientation between frames are JPEG0007595159000024.jpg59 can be expressed as a 6-dimensional twist vector using Lie algebra, i.e. The image is JPEG0007595159000025.jpg661.

[0056] In one implementation scenario, in order to facilitate the subsequent determination of the relevant attributes of each pixel point, a contour mask can be obtained based on the projection result of the target object, and each pixel point in the contour mask corresponds to a pixel point at the same position in the captured image. Referring to FIG. 2, FIG. 2 is a schematic diagram of one embodiment of a contour mask. As shown in FIG. 2, after the target object is projected, a projection contour can be obtained, and the captured image is divided into a foreground region (i.e., the foreground region in FIG. 2) based on the projection contour. JPEG0007595159000026.jpg67) and the background region (i.e., the background region in FIG. 2 JPEG0007595159000027.jpg66).

[0057] In one implementation, the projected contour points on the projected contour Search line that passes through JPEG0007595159000028.jpg56 JPEG0007595159000029.jpg63 can be constructed. The search line segment is specifically the projected contour point Normal vector of projected contour in JPEG0007595159000030.jpg56 Based on this, some sampling points can be extracted along the search line. For example, Projected contour points on JPEG0007595159000032.jpg63 JPEG0007595159000033.jpg56 and projected contour points N pixels (7, 8, 9, etc.) located on both sides of JPEG0007595159000034.jpg56 are searched for These can be extracted as several sampling points (i.e., the black circles in FIG. 2) on JPEG0007595159000035.jpg63.

[0058] It should be understood that what is shown in FIG. 2 is only a projection contour that may exist in an actual application process, and does not limit the specific steps of the projection contour, and such analogies can be made in other cases, and will not be repeated here.

[0059] In one implementation, refer to Fig. 3, which is a schematic diagram of one embodiment of a projected contour and a search line segment. As shown in Fig. 3, in a real scene, a number of search line segments can be constructed based on each projected contour point on the projected contour. This analogy can be applied to other captured images, and will not be described again here.

[0060] In step S12, attribute information of the sampling points is obtained, and reference weights of the sampling points are obtained.

[0061] In the present embodiment, the attribute information represents the possibility that the sampling point belongs to the target object. For example, the attribute information may include a first probability value and a first reliability that the sampling point belongs to the target object, and a second probability value and a second reliability that the sampling point does not belong to the target object.

[0062] It should be understood that the above first reliability represents the reliability of the first probability value, and the above second reliability can represent the reliability of the second probability value. Furthermore, if a sampling point belongs to a target object, the sampling point can be regarded as belonging to an actual foreground region in the captured image, and conversely, if a sampling point does not belong to a target object, the sampling point can be regarded as belonging to an actual background region in the captured image.

[0063] In one implementation, for convenience of explanation, the search line The jth sampling point on JPEG0007595159000036.jpg63 is JPEG0007595159000037.jpg66, and the sampling point The first probability value for JPEG0007595159000038.jpg66 is JPEG0007595159000039.jpg815, and the sampling point The second probability value for JPEG0007595159000040.jpg66 is It can be written as JPEG0007595159000041.jpg815.

[0064] It should be understood that the first probability value and the second probability value can be determined by a time-series local color histogram, and the specific acquisition process of the first probability value and the second probability value can be referred to the specific technical details of the time-series local color histogram, and will not be repeated here.

[0065] In one implementation, the first reliability is negatively correlated with the second reliability, i.e., the higher the first reliability, the lower the second reliability, and conversely, the lower the first reliability, the higher the second reliability. Furthermore, the first reliability of a sampling point is negatively correlated with the oriented Euclidean distance from the corresponding projected contour point to the sampling point, and the corresponding projected contour point and the sampling point are located on the same search line segment. Referring to FIG. 2, the search line segment For example, in JPEG0007595159000042.jpg63, the search line The first reliability of a particular sampling point on JPEG0007595159000043.jpg63 is calculated by the search line segment Projected contour points on JPEG0007595159000044.jpg63 There is a negative correlation with the oriented Euclidean distance to JPEG0007595159000045.jpg56, and for each sampling point on the other search line segments, the corresponding first reliability can be obtained in the same manner as above, and the description will not be repeated here.

[0066] In one specific implementation, the directional Euclidean distance from the sampling point to the corresponding projected contour point can be obtained based on the first coordinate of the sampling point, the second coordinate of the corresponding projected contour point, and the normal vector. The jth sampling point on JPEG0007595159000046.jpg63 For example, take JPEG0007595159000047.jpg66. Its oriented Euclidean distance is JPEG0007595159000048.jpg713 can be represented as follows: JPEG0007595159000049.jpg1054…(3)

[0067] In the above formula (3), JPEG0007595159000050.jpg56 is the search line represents the projected contour points on JPEG0007595159000051.jpg63, JPEG0007595159000052.jpg76 is the projected contour point Represents the transpose of the normal vector of the projected contour in JPEG0007595159000053.jpg56.

[0068] In another specific implementation, in order to smooth the first confidence, a smooth and differentiable step function (such as the Heaviside function) can be adopted to process the directed Euclidean distance to obtain the first confidence. The jth sampling point on JPEG0007595159000054.jpg63 For example, JPEG0007595159000055.jpg66 has the highest reliability. JPEG0007595159000056.jpg924 can be represented as follows: JPEG0007595159000057.jpg1592…(4)

[0069] In the above formula (4), JPEG0007595159000058.jpg43 represents the smoothing factor, The larger the JPEG0007595159000059.jpg43, the higher the first reliability. JPEG0007595159000060.jpg924 is the directional Euclidean distance It changes more drastically depending on JPEG0007595159000061.jpg713, and conversely, The smaller JPEG0007595159000062.jpg43 is, the higher the reliability is. JPEG0007595159000063.jpg924 is the directional Euclidean distance The change is more gradual depending on JPEG0007595159000064.jpg713.

[0070] In yet another embodiment, the sum of the first reliability and the second reliability may be 1. The jth sampling point on JPEG0007595159000065.jpg63 For example, JPEG0007595159000066.jpg66 has the following sampling points: JPEG0007595159000067.jpg66 1st reliability After getting JPEG0007595159000068.jpg924, JPEG0007595159000069.jpg933, sampling point It can be used as the second reliability of JPEG0007595159000070.jpg66.

[0071] In one implementation scenario, for each search line segment, a target point can be searched from several sampling points on the search line segment to obtain a search result, and the target point is used to represent an object contour point of the target object. On this basis, weight information of several sampling points on the search line segment can be obtained respectively based on the search result, the weight information includes at least one of a first weight and a second weight, the first weight is correlated with a predicted probability value of the target point, the predicted probability value represents the possibility that the sampling point is an object contour point, and the second weight is correlated with a first distance from the target point to the sampling point, so that a reference weight of the sampling point can be obtained based on the weight information.

[0072] The process of searching for the target point and the process of calculating the predicted probability value can be referred to in the following embodiments, and will not be described again here. Furthermore, the object contour point is the actual contour point of the target object in the captured image, and as shown in FIG. 2, the search line segment Sampling points on JPEG0007595159000071.jpg63 If JPEG0007595159000072.jpg54 is located on the object contour (i.e., the object contour in FIG. 2), the sampling point JPEG0007595159000073.jpg54 is also the actual contour point of the target object in the captured image.

[0073] In one specific implementation scenario, the weight information may include a first weight, and when the search result includes that the target point is searched, the first weight of the sampling point can be determined based on the predicted probability value of the target point, the first weight being positively correlated with the predicted probability value of the target point, and when the search result includes that the target point is not searched, the first weight can be determined as a first numerical value, and the first numerical value is a lower limit value of the first weight when the search result includes that the target point is searched. Still, the search line segment The jth sampling point on JPEG0007595159000074.jpg63 For example, in JPEG0007595159000075.jpg66, the first weight JPEG0007595159000076.jpg918 can be represented as follows: JPEG0007595159000077.jpg19115…(5)

[0074] In the above formula (5), JPEG0007595159000078.jpg716 is the search line JPEG0007595159000079.jpg63 is the target point This means that you can search for JPEG0007595159000080.jpg54. JPEG0007595159000081.jpg720 is the target point Predicted probability value of JPEG0007595159000082.jpg54, i.e., the target point JPEG0007595159000083.jpg54 represents the possibility of being an object contour point, and the target point The higher the predicted probability value of JPEG0007595159000084.jpg54, the closer the target point is to the target point. JPEG0007595159000085.jpg54 is more likely to be an object contour point, and conversely, the target point The smaller the predicted probability value of JPEG0007595159000086.jpg54, the closer the target point is to the target point. The possibility that JPEG0007595159000087.jpg54 is an object contour point is reduced. JPEG0007595159000088.jpg75 represents a negative constant, which is used to control the decay rate of the first weight which decays according to the predicted probability value. Specifically, it can be set according to the requirements of the application, for example, it can be set to -1.25, etc., and is not limited here.

[0075] As shown in equation (5), the target point The predicted probability value of JPEG0007595159000089.jpg54 is 1, which means that the target point JPEG0007595159000090.jpg54 is the most likely object contour point. In this case, the target point JPEG0007595159000091.jpg54 has the largest first weight (i.e., 1) and is the target point The predicted probability value of JPEG0007595159000092.jpg54 is 0, which means that the target point JPEG0007595159000093.jpg54 is the least likely object contour point. In this case, the target point JPEG0007595159000094.jpg54 has the smallest first weight (i.e., JPEG0007595159000095.jpg718), and the first weighting is a lower limit value of the first weighting when the search result includes that the target point has been searched.

[0076] target point Predicted probability value for JPEG0007595159000096.jpg54 If JPEG0007595159000097.jpg720 is too small, the target point Search line where JPEG0007595159000098.jpg54 is located JPEG0007595159000099.jpg63 is likely to be in an interference state (locally occluded, in a local area of ​​interference by similar colors, etc.), in which case the target point Search line where JPEG0007595159000100.jpg54 is located It should be understood that by reducing the first weight of each sampling point on JPEG0007595159000101.jpg63, the reference value of these sampling points in the subsequent position and orientation parameter acquisition process can be reduced, thereby reducing the influence of interference factors on the position and orientation parameters as much as possible and improving the accuracy of the position and orientation parameters.

[0077] In another specific implementation scenario, the weight information may include a second weight, and when the search result includes that the target point is searched, the second weight of the sampling point can be determined based on the first distance corresponding to the sampling point, and the second weight is negatively correlated with the first distance, and when the search result includes that the target point is not searched, the second weight can be determined as a second numerical value, and the second numerical value is an upper limit value of the second weight when the search result includes that the target point is not searched. The jth sampling point on JPEG0007595159000102.jpg63 For example, in JPEG0007595159000103.jpg66, the second weight JPEG0007595159000104.jpg1020 can be represented as follows: JPEG0007595159000105.jpg20109…(6)

[0078] In the above formula (6), JPEG0007595159000106.jpg716 is the search line JPEG0007595159000107.jpg63 is the target point This means that you can search for JPEG0007595159000108.jpg54. JPEG0007595159000109.jpg1025 is the sampling point The first distance corresponding to JPEG0007595159000110.jpg66, i.e. the target point Sampling points from JPEG0007595159000111.jpg54 It represents the distance to JPEG0007595159000112.jpg66. Specifically, the first distance JPEG0007595159000113.jpg1025 is the sampling point First coordinate of JPEG0007595159000114.jpg66, target point The third coordinate and sampling point of JPEG0007595159000115.jpg54 Search line where JPEG0007595159000116.jpg66 is located Length of JPEG0007595159000117.jpg63 JPEG0007595159000118.jpg76 (i.e., search line JPEG0007595159000119.jpg63

[0079] The number of sampling points in can be calculated based on the number of sampling points in, for example, The image is JPEG0007595159000120.jpg957.

[0080] Furthermore, JPEG0007595159000121.jpg65 represents a negative constant, which is used to control the decay rate of the second weight that decays according to the first distance. Specifically, it can be set according to the requirements of the application, for example, -3.5, etc., but is not limited here. As shown in Equation (6), the sampling point JPEG0007595159000122.jpg66 and the target point If JPEG0007595159000123.jpg54 are the same sampling points, the first distance between them JPEG0007595159000124.jpg1025 reaches the minimum value (i.e. 0).

[0081] In this case, the target point JPEG0007595159000125.jpg54 has the largest second weight (i.e., 1) and the sampling point JPEG0007595159000126.jpg66 and the target point JPEG0007595159000127.jpg54 is the search line If they are located at both ends of JPEG0007595159000128.jpg63, the first distance between them JPEG0007595159000129.jpg1025 reaches the maximum value (i.e., 1), and in this case the target point JPEG0007595159000130.jpg54 has the smallest second weight (i.e., JPEG0007595159000131.jpg821).

[0082] Sampling points JPEG0007595159000132.jpg66 is the target point The further away from JPEG0007595159000133.jpg54, the more sampling points JPEG0007595159000134.jpg66 is prone to interference (interference due to a complex background, interference due to similar colors, etc.), and in this case, the sampling point By reducing the second weight of JPEG0007595159000135.jpg66, the sampling point in the subsequent process of acquiring position and orientation parameters is It should be understood that the reference value of JPEG0007595159000136.jpg66 can be reduced, thereby reducing the influence of interference factors on the position and orientation parameters as much as possible and improving the accuracy of the position and orientation parameters.

[0083] In yet another specific implementation, the weight information of the sampling point may include both the first weight and the second weight, in this case, the obtaining process of the first weight and the second weight can refer to the obtaining method of the first weight and the second weight respectively described above, and will not be described again here.Furthermore, when the weight information of the sampling point includes both the first weight and the second weight, both the first weight and the second weight are positively correlated with the reference weight.

[0084] For example, the product of the first weight and the second weight can be used as the reference weight. The jth sampling point on JPEG0007595159000137.jpg63 For example, in JPEG0007595159000138.jpg66, the first weight JPEG0007595159000139.jpg1019 and the second weight The product of JPEG0007595159000140.jpg1019 is the sampling point Reference weight for JPEG0007595159000141.jpg66 The same applies to the other sampling points, and will not be described here again.

[0085] In step S13, an objective function is constructed based on the attribute information of the sampling points and the reference weights.

[0086] In one implementation scenario, as described above, the attribute information of the sampling points may include a first probability value and a first reliability that the sampling point belongs to the target object, and a second probability value and a second reliability that the sampling point does not belong to the target object. In this case, a first product of the first reliability and the first probability value, and a second product of the second reliability and the second probability value are obtained, and the joint probability of the sampling points can be obtained based on the first product and the second product. Based on this, the objective function is then obtained based on the weighting result of the joint probability of each sampling point by the reference weight.

[0087] In one particular implementation, the joint probability is obtained by taking the logarithm of the sum of the first product and the second product, and the joint probability is weighted by the reference weights of each sampling point and summed to obtain the objective function Get JPEG0007595159000143.jpg714. JPEG0007595159000144.jpg11123…(7)

[0088] In the above formula (7), JPEG0007595159000145.jpg923 represents the first reliability, JPEG0007595159000146.jpg816 represents the first probability value, JPEG0007595159000147.jpg930 represents the second reliability, JPEG0007595159000148.jpg916 represents the second probability value, JPEG0007595159000149.jpg916 represents the reference weight, and the specific acquisition process can be referred to the above related description, which will not be described here repeatedly. JPEG0007595159000150.jpg54 represents a set of sampling points on all the search line segments.

[0089] In another specific embodiment, referring to FIG. 2, the captured image is divided into foreground regions (i.e., the foreground regions in FIG. 2) based on the projected contours (i.e., the projected contours in FIG. 2). JPEG0007595159000151.jpg67) and the background region (i.e., the background region in FIG. 2 JPEG0007595159000152.jpg66), and before constructing the objective function, each sampling point on the search line segment can be verified. For example, if the direction of the normal vector at the projected contour point is from the foreground region to the background region, if the oriented Euclidean distance of the sampling point is greater than a first distance value (such as 0), the sampling point can be considered to belong to the background region.

[0090] If the sampling point actually belongs to the foreground region, the sampling point can be filtered, that is, the sampling point can be filtered from the above sampling point set L. As shown in the bottom search line segment in Figure 2, the directed Euclidean distances of the two sampling points located at the leftmost end of the search line segment are both greater than a first distance value (such as 0), so these two sampling points can be considered to belong to the background region. If these two sampling points actually belong to the foreground region, these two sampling points can be filtered.

[0091] Similarly, if the directional Euclidean distance of a sampling point is smaller than a second distance value (such as 0), the sampling point can be considered to belong to the foreground region, and if some sampling points actually belong to the background region, they can be filtered out; that is, from the above sampling point set L, the sampling points belonging to the background region can be filtered out. As shown in the search line segment in the upper left corner of Figure 2, the directional Euclidean distances of the two sampling points located at the rightmost end of the search line segment are both smaller than the second distance value (such as 0), so these two sampling points can be considered to belong to the foreground region, and if these two sampling points actually belong to the background region, these two sampling points can be filtered out.

[0092] In step S14, the position and orientation parameters of the target object in the captured image are obtained based on the objective function.

[0093] In one implementation scenario, as described above, the projected contour is obtained by projecting using a reference pose of the target object, where the reference pose is the pose parameters of the target object in a reference image, which is captured before the captured image. In this case, the objective function can be solved to obtain update parameters of the reference pose, and the update parameters can be used to optimize the reference pose to obtain the pose parameters of the target object in the captured image.

[0094] For example, the reference image may be an image of the frame immediately before the captured image, for example, the reference image may be an image of the t-1th frame in the video data, and the captured image may be an image of the tth frame in the video data, for details, please refer to the relevant description above, and the description will not be repeated here.

[0095] In one particular implementation, to facilitate minimization of the objective function by a nonlinear algorithm, the objective function can be rewritten into a standard form of a nonlinear weighted least-squares problem. JPEG0007595159000153.jpg1798…(8) JPEG0007595159000154.jpg6123…(9)

[0096] In the above formula (8), Based on this, the optimization problem can be solved by iterating through the Gauss-Newton method, and the Jacobi vector is defined as follows: JPEG0007595159000156.jpg1979…(10) JPEG0007595159000157.jpg18126…(11) JPEG0007595159000158.jpg19103…(12)

[0097] In the above formula (12), JPEG0007595159000159.jpg1436 represents a smooth Dirac delta function, with the first reliability It can be derived from JPEG0007595159000160.jpg829, and for details, refer to the above-mentioned formula (4). JPEG0007595159000161.jpg1611 can be derived by the above formula (2). For the specific derivation process, please refer to the relevant details of the Gauss-Newton method, and will not be repeated here. Based on the above Jacobian vector and the Gauss-Newton method, the update parameter JPEG0007595159000162.jpg57 can be derived. JPEG0007595159000163.jpg21108…(13)

[0098] In another specific implementation, the above update parameters JPEG0007595159000164.jpg57 is expressed as a Lie algebra, and in order to facilitate the optimization of the reference pose, JPEG0007595159000165.jpg57 to Euclidean transformation matrix JPEG0007595159000166.jpg48. For the specific conversion process, refer to the relevant technical details of Lie groups and Lie algebras, which will not be described here. Based on this, the position and orientation parameters of the target object in the captured image are JPEG0007595159000167.jpg45 can be represented as follows: JPEG0007595159000168.jpg551

[0099] In one embodiment, when there are multiple target objects, if the multiple target objects do not occlude each other, the position and orientation parameters of each target object in the captured image can be obtained by the steps in the embodiment of the present application. Conversely, if the multiple target objects are occluded by each other, the above-mentioned mask image JPEG0007595159000169.jpg65 and depth image JPEG0007595159000170.jpg65 can be used to filter out sampling points that are in a interfered state (e.g., occluded).

[0100] In one particular implementation, the depth image JPEG0007595159000171.jpg65 may be obtained by rendering the captured image, and the specific rendering process is not described in detail here. JPEG0007595159000172.jpg65 may specifically include the depth value of each pixel point of the captured image.

[0101] In another specific embodiment, in the process of obtaining the position and orientation parameters of the k-th target object in the captured image, first, a background region corresponding to the k-th target object is obtained. Projected contour points of the search line segment located within JPEG0007595159000173.jpg77 Adjacent sampling points to JPEG0007595159000174.jpg56 You can search for JPEG0007595159000175.jpg66, Whether JPEG0007595159000176.jpg916 is equal to another target object index and sampling point Depth value of JPEG0007595159000177.jpg66 JPEG0007595159000178.jpg916 is the sampling point Projected contour point of the search line segment where JPEG0007595159000179.jpg66 is located Depth value of JPEG0007595159000180.jpg56 Check whether it is smaller than JPEG0007595159000181.jpg817, and if so, check the sampling point Search line where JPEG0007595159000182.jpg66 is located JPEG0007595159000183.jpg73 can be considered to be in an interference state (e.g., occluded), and this search line All sampling points on JPEG0007595159000184.jpg73 can be filtered.

[0102] Referring to Figures 4a to 4c, where Figure 4a is a schematic diagram of one embodiment of a captured image, Figure 4b is a schematic diagram of another embodiment of a mask image, and Figure 4c is a schematic diagram of one embodiment of a search line segment.

[0103] As shown in Figures 4a to 4c, there are two target objects, a duck and a squirrel, in the captured image, and the duck is occluded by the squirrel. Therefore, in the process of obtaining the position and orientation parameters of the duck in the captured image, the search line segments near the occluded projected contour can be filtered to reduce the adverse effects of interference factors such as local occlusion on the position and orientation parameters as much as possible, thereby improving the accuracy of the position and orientation parameters.

[0104] In one implementation, the aforementioned local color histogram is constructed and obtained based on local regions surrounding the object contour, and in order to enhance temporal continuity, each local region corresponds to one model vertex. However, when the 3D model of the target object contains a small number of vertices (e.g., less than 50), these local regions cannot completely cover the object contour, so the above first probability value JPEG0007595159000185.jpg65 and the second probability value This affects the accuracy of JPEG0007595159000186.jpg65.

[0105] In view of the above problems, if the number of vertices contained in the 3D model of the target object is less than a predetermined threshold (e.g., 50), some vertices (e.g., four) can be added to each side of the 3D model to increase the number of local regions, thereby enabling the local regions to cover the object contour as much as possible.

[0106] Referring to Figures 5a to 5c, where Figure 5a is a schematic diagram of another example of a captured image, Figure 5b is a schematic diagram of one example of a layout area, and Figure 5c is a schematic diagram of another example of a local area.

[0107] As shown in Figures 5a to 5c, when the number of vertices (indicated by black circles in the figures) is relatively small (for example, there are only four in Figure 5b), the local region (indicated by white circles in the figures) does not completely cover the object contour of the target object. In this case, adding vertices to each side (e.g., adding to eight) allows the local region to completely cover the object contour.

[0108] In one implementation, the position and orientation parameters of the target object in the captured image may be obtained by multiple iterations (e.g., seven iterations). Specifically, in the first iteration process, the position and orientation parameters of the target object in the reference image are used as a reference position to execute the steps in the embodiment of the present application to obtain the position and orientation parameters of the target object in the captured image in the first iteration, and the steps in the embodiment of the present application are used as a reference position and orientation for the second iteration to execute the steps in the embodiment of the present application again to obtain the position and orientation parameters of the target object in the captured image in the second iteration.

[0109] By this analogy, in the i-th iteration, the position and orientation parameters acquired in the (i-1)-th iteration are used as reference position and orientation to execute the steps in the embodiment of the present application to acquire the position and orientation parameters of the target object in the captured image in the i-th iteration, and this process is repeated until the final iteration, and the position and orientation parameters of the target object in the captured image in the final iteration can be directly used as the final position and orientation parameters of the target object in the captured image.

[0110] In the above technical solution, several sampling points located on a search line segment in the captured image are obtained, the search line segment passes through the projected contour points of the target object in the captured image, and the projected contour points are located on the projected contour of the target object. Based on this, attribute information of the sampling points is then obtained, and reference weights of the sampling points are obtained, where the attribute information represents the possibility that the sampling points belong to the target object. Thus, an objective function is constructed based on the attribute information and reference weights of the sampling points, and the position and orientation parameters of the target object in the captured image are obtained based on the objective function.

[0111] It should be understood that the objective function is constructed based on both the attribute information of the sampling points and the reference weights, so that, on the one hand, the attribute information can indicate the possibility that the sampling points belong to the target object, and, on the other hand, the reference weights can indicate the reference value of the sampling points in the subsequent process of solving the position and attitude parameters, thereby making it possible to reduce the influence of interference factors on the position and attitude solving as much as possible and improve the accuracy of the position and attitude parameters.

[0112] Referring to Figure 6, Figure 6 is a flowchart of one embodiment of step S12 in Figure 1. Specifically, it may include the following steps.

[0113] In step S61, a target point is searched for from among several sampling points on a search line segment, and a search result is obtained.

[0114] In the present embodiment, the target points are used to represent the object contour points of the target object. Referring to FIG. 2, the search line segment For example, in JPEG0007595159000187.jpg63, the search line Sampling points on JPEG0007595159000188.jpg63 JPEG0007595159000189.jpg65 may be used to represent object contour points, as well as other cases, which will not be repeated here.

[0115] In one implementation scenario, for each search line segment, several sampling points on the search line segment can be used as the current point, and if the reference probability difference of the current point satisfies the first condition, the current point can be used as the candidate point, and the candidate point whose predicted cost value satisfies the second condition can be selected as the target point. According to the above method, a rough selection is first performed based on the reference probability difference, and then a precise selection is performed based on the predicted cost value, thereby improving the selection efficiency and accuracy of the target point.

[0116] In one specific implementation scenario, the reference probability difference of the current point can be the difference between the first probability values ​​of two sampling points that have a certain positional relationship with the current point (such as adjacent to the current point). In order to improve the convenience of accessing the first probability values ​​of different sampling points, as shown in FIG. 7a, all the search line segments in FIG. 3 are stacked row by row to form a cluster image of the search line segments. JPEG0007595159000190.jpg64 can be constructed, and the first probability values ​​of each sampling point on all the search line segments in FIG. 3 are also stacked row by row to construct a cluster image for the first probability values, as shown in FIG. 7b. JPEG0007595159000191.jpg75 can be constructed.

[0117] As described in the previous embodiment, the search line segment includes 2*N+1 sampling points, where the sampling point at the middle position is a projected contour point, one side of the projected contour point corresponds to the foreground region, and the other side of the projected contour point corresponds to the background region, so that the cluster image The center row of JPEG0007595159000192.jpg64 corresponds to the projected contour, one side of the center row corresponds to the foreground region, and the other side of the center row corresponds to the background region.

[0118] In another particular implementation, the first condition may include a reference probability difference being greater than a predetermined threshold, and the cluster image After constructing and obtaining JPEG0007595159000193.jpg75, we apply the given convolution kernel ( JPEG0007595159000194.jpg628) to create a cluster image A sliding convolution can be performed on each of the 75 rows of JPEG0007595159000195.jpg, and the convolution response is calculated based on the JPEG0007595159000196.jpg44 A sampling point higher than (eg, 0.3) can be used as a candidate point.

[0119] In yet another specific implementation scenario, there may be three situations for the above candidate points: the candidate points are object contour points, the candidate points are foreground interference points, and the candidate points are background interference points. The jth sampling point on JPEG0007595159000197.jpg63 For JPEG0007595159000198.jpg65, the search line points to the background area Select some consecutive sampling points from a portion of JPEG0007595159000199.jpg63 to create the first sampling point set. JPEG0007595159000200.jpg89 (for example, JPEG0007595159000201.jpg639) and a search segment pointing to the foreground region Select some consecutive sampling points from a portion of JPEG0007595159000202.jpg63 to create a second sampling point set. JPEG0007595159000203.jpg99 (for example, JPEG0007595159000204.jpg641) can be formed.

[0120] Therefore, if the sampling points are object contour points, the first sampling point set JPEG0007595159000205.jpg89 theoretically belongs to the background region, and the second sampling point set Since JPEG0007595159000206.jpg99 should theoretically belong to the foreground region, the probability that the sampling point is an object contour point is JPEG0007595159000207.jpg923 can be represented as follows: JPEG0007595159000208.jpg21124…(15)

[0121] If JPEG0007595159000209.jpg75 is a candidate point, It should be understood that JPEG0007595159000210.jpg923 is a predicted probability value that the candidate point is an object contour point. JPEG0007595159000211.jpg75 is the target point If it is available as JPEG0007595159000212.jpg65, JPEG0007595159000213.jpg923 is JPEG0007595159000214.jpg721, that is, the predicted probability value that the target point is an object contour point.

[0122] Similarly, if the sampling point is a foreground interference point, the first sampling point set JPEG0007595159000215.jpg89 and the second sampling point set Theoretically, both of the JPEG0007595159000216.jpg99 should belong to the foreground region, so the probability that the sampling point is a foreground interference point is JPEG0007595159000217.jpg922 can be represented as follows: JPEG0007595159000218.jpg19116…(16)

[0123] If JPEG0007595159000219.jpg75 is a candidate point, It should be understood that JPEG0007595159000220.jpg923 is a predicted probability value that the candidate point is a foreground interference point.

[0124] Similarly, if the sampling point is a background interference point, the first sampling point set JPEG0007595159000221.jpg89 and the second sampling point set Theoretically, both of the JPEG0007595159000222.jpg99 should belong to the background region, so the probability that the sampling point is a background interference point is JPEG0007595159000223.jpg923 can be represented as follows: JPEG0007595159000224.jpg21120…(17)

[0125] If JPEG0007595159000225.jpg75 is a candidate point, It should be understood that JPEG0007595159000226.jpg923 is the predicted probability value that the candidate point is a background interference point.

[0126] Based on this, the normalized probability value that the sampling point is an object contour point is JPEG0007595159000227.jpg917

[0127] can be further defined. JPEG0007595159000228.jpg21112…(18)

[0128] If JPEG0007595159000229.jpg75 is a candidate point, It should be understood that JPEG0007595159000230.jpg916 is a normalized probability value that the candidate point is an object contour point.

[0129] In yet another specific implementation, the predicted probability that the candidate point is an object contour point is After obtaining JPEG0007595159000231.jpg922, or the normalized probability that the candidate point is an object contour point After getting JPEG0007595159000232.jpg917, the predicted probability value The candidate points for which JPEG0007595159000233.jpg922 satisfies the third condition can be further filtered. For example, the predicted probability value JPEG0007595159000234.jpg922 is the aforementioned probability value JPEG0007595159000235.jpg922 and probability value Candidate points smaller than the maximum value of JPEG0007595159000236.jpg922 can be filtered out, i.e., candidate points In the case of JPEG0007595159000237.jpg75, Candidate points that satisfy JPEG0007595159000238.jpg782 JPEG0007595159000239.jpg75 can be filtered, or, as mentioned above, the predicted probability value Normalized probability value based on JPEG0007595159000240.jpg922 Get JPEG0007595159000241.jpg917 and normalize the probability value Candidate points where JPEG0007595159000242.jpg917 is smaller than a predetermined threshold (such as 0.5), which is not limited here, can be filtered.

[0130] In yet another specific implementation, the predicted cost value may include at least one of a first cost value and a second cost value, for example, may include both the first cost value and the second cost value, may include only the first cost value, or may include only the second cost value. The first cost value may be correlated with the predicted probability value of the candidate point, for example, the first cost value may be negatively correlated with the predicted probability value of the candidate point, and for convenience of explanation, the first cost value may be It can be written as JPEG0007595159000243.jpg917, and the first cost value JPEG0007595159000244.jpg917 can be represented as follows: JPEG0007595159000245.jpg17117…(19)

[0131] As shown in equation (19), the candidate points Predicted probability value of JPEG0007595159000246.jpg75 The larger JPEG0007595159000247.jpg922 is, the more likely the candidate point is JPEG0007595159000248.jpg75 is the first cost value for the target point JPEG0007595159000249.jpg917 will be smaller.

[0132] Furthermore, the second cost value is correlated with a second distance from the candidate point to the projected contour point on the search line segment. For example, the second cost value may be positively correlated with the above second distance. For convenience of explanation, the second cost value is It can be written as JPEG0007595159000250.jpg717, and the second cost value JPEG0007595159000251.jpg717 can be represented as follows: JPEG0007595159000252.jpg1359…(20)

[0133] As shown in equation (20), the candidate points Search line from JPEG0007595159000253.jpg75 Projected contour points on JPEG0007595159000254.jpg73 Second distance to JPEG0007595159000255.jpg56 The larger JPEG0007595159000256.jpg1021 is, the more likely the candidate point is JPEG0007595159000257.jpg75 is the target point, the second cost value JPEG0007595159000258.jpg717 will be larger.

[0134] If the predicted cost value includes both the first cost value and the second cost value, a weighting process is performed on the first cost value and the second cost value to obtain a predicted cost value. It should be understood that the image can be used as JPEG0007595159000259.jpg716. JPEG0007595159000260.jpg989…(21)

[0135] In the above formula (21), JPEG0007595159000261.jpg55 represents a weighting factor, which can be specifically set according to actual applications, for example, 0.015, etc., and is not limited here. The above second condition can specifically include that the predicted cost value is the smallest, that is, when the frame-to-frame motion of the target object or the camera is smooth, the second cost value gives an additional penalty to the candidate point far away from the projected contour, and preferentially selects the candidate point close to the projected contour as the target point.

[0136] In the above selection process, the search line There is a possibility that the target point is not found on JPEG0007595159000262.jpg73. In this case, Regarding JPEG0007595159000263.jpg73, Mark it as JPEG0007595159000264.jpg615 to search for the line segment It should be appreciated that the search results for JPEG0007595159000265.jpg73 indicate that no landmarks are found.

[0137] In step S62, weight information for each of several sampling points on the search line segment is obtained based on the search result.

[0138] In the present embodiment, the weight information includes at least one of a first weight and a second weight, the first weight is correlated with a predicted probability value of the target point, the predicted probability value represents the possibility that the sampling point is an object contour point, and the second weight is correlated with a first distance from the target point to the sampling point. For details, please refer to the relevant description of the above embodiment, and will not be repeated here.

[0139] In step S63, the reference weights of the sampling points are obtained based on the weight information.

[0140] For details, please refer to the relevant descriptions in the above embodiments, and the description will not be repeated here.

[0141] In the above technical solution, a target point is searched for from several sampling points on a search line segment to obtain a search result, the target point is used to represent an object contour point of the target object, and weight information of several sampling points on the search line segment is obtained respectively based on the search result, the weight information includes at least one of a first weight and a second weight, the first weight is correlated with a predicted probability value of the target point, and the predicted probability value represents the possibility that the sampling point is an object contour point, and the second weight is correlated with a first distance from the target point to the sampling point, so that the first weight and the second weight can represent the reference value of the sampling point from different angles, and based on this, the reference weight of the sampling point is then obtained based on the weight information, so that the reference value of the reference weight in the subsequent process of solving the position and posture parameters can be improved.

[0142] 8, which is a flowchart of another embodiment of a position and orientation acquisition method in the present application. Specifically, the method may include the following steps:

[0143] In step S801, the captured image is downsampled to obtain pyramid images with several resolutions.

[0144] For example, downsampling can be performed on the captured image with a downsampling ratio of 2 to obtain a pyramid image with 1 / 4 resolution, a pyramid image with 1 / 2 resolution, and a pyramid image with the original resolution (i.e., the captured image itself).The same is true for other cases, and will not be described again here.

[0145] In step S802, pyramid images are sequentially selected according to resolution from small to large as the current captured image.

[0146] For example, a 1 / 4 resolution pyramid image can be selected as the current captured image, and the steps described below can be performed to obtain the position and orientation parameters of the target object in the 1 / 4 resolution pyramid image, and then a 1 / 2 resolution pyramid image can be selected as the current captured image, and so on, which will not be repeated here.

[0147] In step S803, several sampling points located on the search line segment in the current captured image are obtained.

[0148] In the embodiment of the present invention, the search line segment passes through the projected contour points of the target object in the captured image, the projected contour points are located on the projected contour of the target object, and the projected contour is obtained by projecting the reference position and orientation of the target object. For details, please refer to the relevant descriptions in the above embodiments, and will not be repeated here.

[0149] In step S804, the attribute information of the sampling points is obtained, and the reference weights of the sampling points are obtained.

[0150] In the embodiment of the present invention, the attribute information represents the probability that the sampling point belongs to the target object. For details, please refer to the relevant description of the above embodiment, and will not be described again here.

[0151] The attribute information includes a first probability value and a first reliability that the sampling point belongs to the target object, and a second probability value and a second reliability that the sampling point does not belong to the target object, and the calculation process of the first reliability can be referred to the formula (4) and its related description in the above embodiment, and it should be understood that the first reliability is negatively correlated with the smoothing coefficient s in the formula (4). For example, for a pyramid image with 1 / 4 resolution, the smoothing coefficient s can be set to 1.2, for a pyramid image with 1 / 2 resolution, the smoothing coefficient s can be set to 0.8, and for a pyramid image with the original resolution (i.e., the captured image itself), the smoothing coefficient s can be set to 0.6, and is not limited here.

[0152] In step S805, an objective function is constructed based on the attribute information of the sampling points and the reference weights.

[0153] For details, please refer to the relevant descriptions in the above embodiments, and the description will not be repeated here.

[0154] In step S806, the position and orientation parameters of the target object in the current captured image are obtained based on the objective function.

[0155] For details, please refer to the relevant descriptions in the above embodiments, and the description will not be repeated here.

[0156] As described in the above embodiment, it should be understood that for a captured image, the position and orientation parameters of the target object in the captured image are obtained by multiple iterations. Similarly, for a pyramid image, the position and orientation parameters of the target object in the pyramid image can be obtained by several iterations, and the lower the resolution of the pyramid image, the more the number of iterations. For example, for a pyramid image with 1 / 4 resolution, it can be iterated four times, for a pyramid image with 1 / 2 resolution, it can be iterated two times, and for a pyramid image with the original resolution (i.e., the captured image itself), it can be iterated once. For the specific iteration process, please refer to the relevant description of the above embodiment, and will not be repeated here.

[0157] In step S807, it is determined whether the currently captured image is the pyramid image of the last frame. If the currently captured image is not the pyramid image of the last frame, step S808 is executed; if the currently captured image is the pyramid image of the last frame, step S810 is executed.

[0158] If the current captured image is the last frame pyramid image, the position and orientation parameters of the target object in the last frame pyramid image can be used as the final position and orientation parameters of the target object in the captured image; otherwise, the iterative process can continue.

[0159] In step S808, the position and orientation parameters acquired in the current execution are used as the reference position and orientation.

[0160] If the current captured image is not the last frame pyramid image, the pose parameters obtained in this run can be used as the reference pose to perform the iterative operation on the next frame pyramid image.

[0161] In step S809, steps S802 and after are executed again.

[0162] That is, if the currently captured image is not the pyramid image of the last frame, the iterative operation is performed on the pyramid image of the next frame.

[0163] In step S810, the position and orientation parameters acquired in the current execution are used as the final position and orientation parameters of the target object in the captured image.

[0164] If the current captured image is the last frame pyramid image, the iterative operation can be terminated to obtain the final position and orientation parameters of the target object in the captured image.

[0165] In the above technical solution, the projected contour is obtained by projecting using the reference position and orientation of the target object, whereby, before the projection sampling, the captured image is first downsampled to obtain pyramid images of several resolutions, and the pyramid images are sequentially selected as the current captured image according to the resolution from small to large, and the steps of obtaining several sampling points located on the search line segment in the above captured image for the current captured image are executed, and the reference position and orientation adopted in this execution are the position and orientation parameters obtained in the previous execution, and the position and orientation parameters obtained in the last execution are used as the final position and orientation parameters of the target object in the captured image, so that in the process of obtaining the position and orientation parameters, the position and orientation estimation can be performed in the order from rough position and orientation estimation to precise position and orientation estimation, and the efficiency and accuracy of obtaining the position and orientation parameters can be further improved.

[0166] 9, which is a schematic diagram of a framework of an embodiment of a position and orientation acquisition device 90 in the present application. The position and orientation acquisition device 90 is applied to an electronic device, and includes a projection sampling module 91, an information extraction module 92, a function construction module 93, and a position and orientation solving module 94, the projection sampling module 91 is configured to acquire some sampling points located on a search line segment in a captured image, where the search line segment passes through a projected contour point of a target object in the captured image, and the projected contour point is located on the projected contour of the target object, the information extraction module 92 is configured to acquire attribute information of the sampling points and acquire reference weights of the sampling points, where the attribute information represents the possibility that the sampling points belong to the target object, the function construction module 93 is configured to construct an objective function based on the attribute information of the sampling points and the reference weights, and the position and orientation solving module 94 is configured to acquire position and orientation parameters of the target object in the captured image based on the objective function.

[0167] In some embodiments, the information extraction module 92 includes a target point search submodule, a weight information acquisition submodule, and a reference weight acquisition submodule. The target point search submodule is configured to search for a target point from several sampling points on a search line segment to obtain a search result, where the target point is used to represent an object contour point of a target object. The weight information acquisition submodule is configured to respectively acquire weight information of several sampling points on the search line segment based on the search result, where the weight information includes at least one of a first weight and a second weight, where the first weight is correlated with a predicted probability value of the target point, the predicted probability value represents a possibility that the sampling point is an object contour point, and the second weight is correlated with a first distance from the target point to the sampling point. The reference weight acquisition submodule is configured to acquire a reference weight of the sampling point based on the weight information.

[0168] In some embodiments, the attribute information includes a first probability value that the sampling point belongs to the target object, and the target point search submodule includes a current point acquisition unit, a candidate point acquisition unit, and a target point acquisition unit. The current point acquisition unit is configured to use several sampling points as current points for each search line segment, respectively. The candidate point acquisition unit is configured to use the current point as a candidate point if a reference probability difference of the current point satisfies a first condition. The target point acquisition unit is configured to select a candidate point whose predicted cost value satisfies a second condition as a target point. Here, the reference probability difference of the current point is a difference between first probability values ​​of two of the sampling points having a predetermined positional relationship with the current point, and the predicted cost value includes at least one of a first cost value and a second cost value, the first cost value being correlated with a predicted probability value of the candidate point, and the second cost value being correlated with a second distance from the candidate point to a projected contour point on the search line segment.

[0169] In some embodiments, the target point search sub-module comprises a candidate point filtering unit configured to filter candidate points whose predicted probability values ​​satisfy a third condition.

[0170] In some embodiments, the predetermined positional relationship is adjacent to the current point, and / or the second condition includes the predicted cost value being the smallest, and / or the first cost value is negatively correlated with the predicted probability value of the candidate point and the second cost value is positively correlated with the second distance.

[0171] In some embodiments, the weight information includes a first weight, and the weight information acquisition submodule includes a first determining unit and a second determining unit. The first determining unit is configured to determine a first weight of the sampling point based on a predicted probability value of the target point when the search result includes that the target point is searched, where the first weight is positively correlated with the predicted probability value of the target point. The second determining unit is configured to determine the first weight as a first numerical value when the search result includes that the target point is not searched, where the first numerical value is a lower limit value of the first weight when the search result includes that the target point is searched.

[0172] In some embodiments, the weight information includes a second weight, and the weight information acquisition submodule includes a third determining unit and a fourth determining unit. The third determining unit is configured to determine a second weight of the sampling point based on a first distance corresponding to the sampling point when the search result includes that the target point is searched, where the second weight is negatively correlated with the first distance. The fourth determining unit is configured to determine the second weight as a second numerical value when the search result includes that the target point is not searched, where the second numerical value is an upper limit value of the second weight when the search result includes that the target point is not searched.

[0173] In some embodiments, the weight information includes a first weight and a second weight, both of which are positively correlated with the reference weight.

[0174] In some embodiments, the attribute information includes a first probability value and a first reliability that the sampling point belongs to the target object, and a second probability value and a second reliability that the sampling point does not belong to the target object, and the function construction module 93 includes a joint probability calculation submodule and a joint probability weighting submodule. The joint probability calculation submodule is configured to obtain a first product of the first reliability and the first probability value and a second product of the second reliability and the second probability value, and obtain a joint probability of the sampling point based on a sum of the first product and the second product. The joint probability weighting submodule is configured to obtain an objective function based on a result of weighting the joint probability of each sampling point by a reference weight.

[0175] In some embodiments, the first reliability is negatively correlated with the second reliability, the first reliability of a sampling point is negatively correlated with a directed Euclidean distance from a corresponding projected contour point to the sampling point, and the corresponding projected contour point and the sampling point are located on the same search line segment.

[0176] In some embodiments, the captured image includes a foreground region and a background region divided based on a projected contour, and the function construction module 93 includes a first filtering sub-module and a second filtering sub-module. The first filtering sub-module is configured to filter a sampling point if the oriented Euclidean distance of the sampling point is greater than a first distance value and the sampling point belongs to the foreground region. The second filtering sub-module is configured to filter a sampling point if the oriented Euclidean distance of the sampling point is less than a second distance value and the sampling point belongs to the background region.

[0177] In some embodiments, the projected contour is obtained by projecting using a reference pose of the target object, and the pose acquisition device 90 includes a downsampling module and an image selection module. The downsampling module is configured to downsample the captured image to obtain pyramid images with several resolutions. The image selection module is configured to sequentially select pyramid images according to resolutions from small to large as the current captured image, and the projection sampling module 91, the information extraction module 92, the function construction module 93, and the pose solving module 94 are configured to perform the steps of obtaining several sampling points located on a search line segment in the captured image and subsequent steps for the current captured image, where the reference pose adopted in this run is the pose parameters obtained in the previous run, and the pose parameters obtained in the last run are used as the final pose parameters of the target object in the captured image.

[0178] In some embodiments, the projected contour is obtained by projecting using a reference pose of the target object, the reference pose being pose parameters of the target object in a reference image, the reference image being captured before the captured image, and the pose solving module 94 comprises a function solving submodule and a pose optimization submodule. The function solving submodule is configured to solve an objective function to obtain update parameters of the reference pose. The pose optimization submodule is configured to optimize the reference pose using the update parameters to obtain the pose parameters.

[0179] 10, which is a schematic diagram of a framework of an embodiment of an electronic device 100 in the present application. The electronic device 100 includes a memory 101 and a processor 102 that are coupled to each other, and the processor 102 is configured to execute any step of the embodiment of the position and orientation acquisition method described above by executing a program stored in the memory 101. In one specific implementation, the electronic device 100 may include, but is not limited to, a microcomputer or a server, and may also include, but is not limited to, a mobile device such as a notebook computer or a tablet computer.

[0180] Specifically, the processor 102 is configured to control itself and the memory 101 to execute any of the steps of the above-mentioned position and orientation acquisition method embodiments. The processor 102 is also called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. Furthermore, the processor 102 may be jointly realized by an integrated circuit chip.

[0181] 11, which is a schematic diagram of a framework of one embodiment of a computer-readable storage medium 110 in the present application. The computer-readable storage medium 110 stores processor-executable program instructions 111, which are used to realize any step of the embodiment of the position and orientation acquisition method described above.

[0182] The above technical proposal can reduce the influence of interference factors on position and orientation solution as much as possible and improve the accuracy of position and orientation parameters.

[0183] An embodiment of the present application provides a computer program including computer readable code, which, when executed by an electronic device, causes a processor of the electronic device to execute any one of the embodiments of the position and orientation acquisition method described above.

[0184] In some examples provided in the present application, it should be understood that the disclosed methods and devices can be realized in other ways. For example, the above-described device embodiments are merely exemplary, and the division into modules or units is merely a division of logical functions, and in actual implementation, there are other division methods, for example, the units or components can be integrated or integrated into another system, or some features can be ignored or not implemented. It should be noted that the shown or discussed mutual couplings or direct couplings or communication connections can be realized using some interfaces, and the indirect couplings or communication connections between the devices or units can be electrical, mechanical, or other forms.

[0185] The units described as separate parts may or may not be physically separated, and the parts shown as units may or may not be physical units, and may be located in one place or distributed across a network of units. According to actual needs, some or all of the units therein may be selected to achieve the objective of the technical solution in this embodiment.

[0186] Furthermore, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be realized in the form of hardware or software functional units.

[0187] When an integrated unit is realized in the form of a software function and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the essential part of the technical solution of the present application, i.e., the part that contributes to the prior art, or all or part of the solution of the technology, may be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute all or part of the processes in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as U disks, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks. [Industrial Applicability]

[0188] The present embodiment discloses a position and orientation acquisition method and its device, electronic device, storage medium and program, in which the position and orientation acquisition method includes: acquiring some sampling points located on a search line segment in a captured image, the search line segment passing through a projected contour point of a target object in the captured image, the projected contour point being located on the projected contour of the target object; acquiring attribute information of the sampling points and acquiring reference weights of the sampling points, the attribute information representing the possibility that the sampling points belong to the target object; constructing an objective function based on the attribute information and reference weights of the sampling points; and acquiring position and orientation parameters of the target object in the captured image based on the objective function. The above technical solution can improve the accuracy of the position and orientation parameters.

Claims

1. A position and orientation acquisition method applied to an electronic device, comprising: Obtaining a number of sampling points located on a search line segment in a captured image, the search line segment passing through a projected contour point of a target object in the captured image, the projected contour point being located on the projected contour of the target object; Obtaining attribute information of the sampling points and obtaining reference weights of the sampling points, the attribute information representing the probability that the sampling points belong to the target object; constructing an objective function based on the attribute information of the sampling points and the reference weights; acquiring position and orientation parameters of the target object in the captured image based on the objective function; Obtaining the reference weights of the sampling points includes: Searching for target points from the several sampling points on the search line segment to obtain a search result, the target points being used to represent object contour points of the target object; obtaining weight information of each of the sampling points on the search line segment based on the search result, the weight information including at least one of a first weight and a second weight, the first weight being correlated with a predicted probability value of the target point, the predicted probability value representing a possibility that the sampling point is the object contour point, and the second weight being correlated with a first distance from the target point to the sampling point; and acquiring reference weights for the sampling points based on the weight information.

2. The attribute information includes a first probability value that the sampling point belongs to the target object, and searching for a target point from the several sampling points on the search line segment to obtain a search result is For each of the search segments, using the several sampling points as a current point respectively, and if a reference probability difference of the current point satisfies a first condition, using the current point as a candidate point, and selecting a candidate point whose predicted cost value satisfies a second condition as the target point; the reference probability difference of the current point is a difference between the first probability values ​​of two sampling points having a predetermined positional relationship with the current point, the predicted cost value includes at least one of a first cost value and a second cost value, the first cost value is correlated with a predicted probability value of the candidate point, and the second cost value is correlated with a second distance from the candidate point to a projected contour point on the search line segment; The position and orientation acquisition method according to claim 1 .

3. Before selecting a candidate point whose predicted cost value satisfies a second condition as the target point, the position and orientation acquisition method further comprises: filtering the candidate points whose predicted probability values ​​satisfy a third condition. The position and orientation acquisition method according to claim 2 .

4. The predetermined positional relationship is adjacent to the current point, and / or the second condition includes that the predicted cost value is a minimum; and / or the first cost value is negatively correlated with a predicted probability value of the candidate point and the second cost value is positively correlated with the second distance. The position and orientation acquisition method according to claim 2 .

5. the weight information includes the first weight, and obtaining weight information of the sampling points on the search line segment based on the search result includes: determining a first weight for the sampling point based on a predicted probability value of the target point when the search result includes that the target point is searched, the first weight being positively correlated with the predicted probability value of the target point; and / or determining the first weight as a first numerical value when the search result includes that the target point is not searched, the first numerical value being a lower limit value of the first weight when the search result includes that the target point is searched; or the weight information includes the second weight, and obtaining weight information of each of the sampling points on the search line segment based on the search result includes determining a second weight of the sampling point based on the first distance corresponding to the sampling point when the search result includes that the target point is searched, the second weight being negatively correlated with the first distance, and / or determining the second weight as a second numerical value when the search result includes that the target point is not searched, the second numerical value being an upper limit value of the second weight when the search result includes that the target point is not searched. The position and orientation acquisition method according to claim 1 .

6. the weight information includes a first weight and a second weight, both of the first weight and the second weight being positively correlated with the reference weight; The position and orientation acquisition method according to claim 1 .

7. The attribute information includes a first probability value and a first reliability that the sampling point belongs to the target object, and a second probability value and a second reliability that the sampling point does not belong to the target object, and constructing an objective function based on the attribute information of the sampling point and the reference weights includes: obtaining a first product of the first reliability and the first probability value and a second product of the second reliability and the second probability value, and obtaining a joint probability of the sampling points based on a sum of the first product and the second product; and obtaining an objective function based on a result of weighting the joint probability by a reference weight of each of the sampling points; the first reliability has a negative correlation with the second reliability, the first reliability of the sampling point has a negative correlation with a directed Euclidean distance from the corresponding projected contour point to the sampling point, and the corresponding projected contour point and the sampling point are located on the same search line segment; The position and orientation acquisition method according to claim 1 .

8. The captured image includes a foreground region and a background region divided based on the projected contour, and before acquiring a first product of the first reliability and the first probability value and a second product of the second reliability and the second probability value, the position and orientation acquisition method includes: filtering the sampling point if the oriented Euclidean distance of the sampling point is greater than a first distance value and the sampling point belongs to the foreground region; and / or filtering the sampling point if the oriented Euclidean distance of the sampling point is smaller than a second distance value and the sampling point belongs to the background region. The position and orientation acquisition method according to claim 7 .

9. The projected contour is acquired by projecting using a reference position and orientation of the target object, and before acquiring some sampling points located on a search line segment in the captured image, the position and orientation acquisition method includes: downsampling the captured image to obtain pyramid images of several resolutions; sequentially selecting the pyramid images as the current captured image according to the resolution from small to large, and executing the steps of obtaining several sampling points located on a search line segment in the captured image for the current captured image and subsequent steps, wherein the reference position and orientation adopted in the current execution are the position and orientation parameters obtained in the previous execution, and the position and orientation parameters obtained in the last execution are used as the final position and orientation parameters of the target object in the captured image. The position and orientation acquisition method according to claim 1 .

10. The projected contour is acquired by projecting using a reference position and orientation of the target object, the reference position and orientation being position and orientation parameters of the target object in a reference image, the reference image being captured before the captured image, and acquiring the position and orientation parameters of the target object in the captured image based on the objective function includes: solving the objective function to obtain update parameters for the reference pose; optimizing the reference pose using the update parameters to obtain the pose parameters. The position and orientation acquisition method according to claim 1 .

11. A position and orientation acquisition device applied to an electronic device, a projection sampling module configured to obtain a number of sampling points located on a search line segment in a captured image, the search line segment passing through projected contour points of a target object in the captured image, the projected contour points being located on the projected contour of the target object; an information extraction module configured to obtain attribute information of the sampling points and obtain reference weights of the sampling points, the attribute information representing a probability that the sampling points belong to the target object; a function construction module configured to construct an objective function based on the attribute information of the sampling points and the reference weights; a pose solving module configured to obtain pose parameters of the target object in the captured image based on the objective function; The information extraction module is specifically configured to search for a target point from the several sampling points on the search line segment to obtain a search result, the target point is used to represent an object contour point of the target object, and based on the search result, obtain weight information of the several sampling points on the search line segment respectively, the weight information including at least one of a first weight and a second weight, the first weight being correlated with a predicted probability value of the target point, the predicted probability value representing the possibility that the sampling point is the object contour point, and the second weight being correlated with a first distance from the target point to the sampling point, and obtain a reference weight of the sampling point based on the weight information.

12. An electronic device 11. An electronic device comprising: a memory and a processor coupled to each other, the processor configured to execute the position and orientation acquisition method according to claim 1 by executing program instructions stored in the memory.

13. A computer readable storage medium having stored thereon program instructions for causing a processor to carry out the method of any one of claims 1 to 10.

14. A computer program comprising computer readable code, the computer program causing a processor of an electronic device to execute the method according to claim 1 when the computer readable code is executed in the electronic device.

Citation Information

Patent Citations

  • Position and orientation measurement apparatus and method thereof

    JP2010079452A

  • Information processing apparatus, control method therefor, and program

    JP2016148649A