An attitude estimation method, apparatus, storage medium, and electronic device

Abstract

The application provides a pose estimation method and device, a storage medium and an electronic device, and relates to the field of image processing. The method comprises: acquiring a query image; if the target object is an unknown target, inputting the query image into a large-scale model, extracting a target image corresponding to the target object in the query image based on the large-scale model; determining a reference image similar to the target image from a reference image set, and determining the pose information of the target object according to the pose information of the reference image. Since the process of pose estimation of the target object in the query image does not require point cloud information and depth information, the cost can be reduced, and the efficiency of pose estimation is provided. The method combines offline and online, the online process is simple and clear, and real-time pose estimation capability is provided. The large-scale model technology is used, which has strong generalization, and can estimate the pose of known targets and unknown targets.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically to a pose estimation method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the development of intelligent technologies, smart construction and unmanned operation have become current research hotspots. Currently, for 3D grasping operations such as crane lifting and unloading hooks and agricultural machinery picking fruits and vegetables, vision-based 6D posture recognition of targets such as hooks is the key to achieving unmanned grasping.

[0003] Currently, most pose recognition methods use image, point cloud, and depth information to estimate the pose of a target and obtain pose information, which is costly and involves a cumbersome processing procedure. Summary of the Invention

[0004] To address the existing technical problems, embodiments of this application provide an attitude estimation method, apparatus, storage medium, and electronic device to improve the efficiency of attitude estimation.

[0005] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an attitude estimation method, the method comprising:

[0007] Obtain a query image, which is used to query the pose information of a target object;

[0008] If the target object is an unknown target, the query image is input into a large-scale model, and the target image corresponding to the target object in the query image is extracted based on the large-scale model;

[0009] Determine reference images similar to the target image from the set of reference images;

[0010] The pose information of the target object is determined based on the pose information of the reference image.

[0011] In this method, after extracting the target image corresponding to the target object from the query image, the present application's embodiments determine a reference image similar to the target image from the reference image set, and determine the target object's pose information based on the pose information of the determined reference image. Thus, the point cloud information or depth information of the target object is not required during the pose information determination process, thereby reducing costs and improving the generalization and efficiency of pose estimation. Furthermore, this application employs large-scale model technology, which has extremely strong generalization capabilities, enabling pose estimation for both known and unknown targets.

[0012] An optional implementation includes inputting the query image into a large-scale model and extracting the target image corresponding to the target object in the query image based on the large-scale model, comprising:

[0013] Obtain the text information corresponding to the target object;

[0014] The text information and the query image are input into the large-scale model, and the target detection is performed on the query image based on the large-scale model to determine the coordinate bounding box of the target object in the query image;

[0015] Extract the image region corresponding to the coordinate frame in the query image, and use the image region as the target image.

[0016] Through the above implementation methods, after determining that the target object is an unknown target, the present application embodiment obtains the text information corresponding to the target, and based on the text information, uses large-scale model technology to perform target detection on the query image, thereby realizing the detection of unknown targets and improving the generalization of pose estimation.

[0017] An optional implementation is that the large-scale model includes a large-scale image-text model and a large-scale object segmentation model; the step of inputting the text information and the query image into the large-scale model, performing object detection on the query image based on the large-scale model, and determining the coordinate bounding box of the target object in the query image includes:

[0018] The text information and the query image are input into the large-scale image-text model, and feature extraction is performed on the query image based on the large-scale image-text model to determine the feature region of the target object;

[0019] The feature region is input into the large-scale object segmentation model, and the feature region is segmented based on the large-scale object segmentation model to obtain the coordinate box of the target object.

[0020] Through the above implementation methods, the embodiments of this application can input text information and query images into a large-scale image and text model, extract features from the query image, determine the approximate location of the target object, and then perform precise segmentation through a large-scale object segmentation model to obtain the precise coordinate frame of the target object.

[0021] An optional implementation of the method further includes:

[0022] If the target object is a known target, the query image is input into the target detector, and the target detection is performed on the query image based on the target detector to determine the coordinate bounding box of the target object in the query image;

[0023] Extract the image region corresponding to the coordinate frame in the query image, and use the image region as the target image.

[0024] Through the above implementation methods, after determining that the target is a known target, the present application embodiment performs target detection on the query image through a first target detector to obtain the target image, which can improve the accuracy and efficiency of target detection.

[0025] An optional implementation includes determining a reference image similar to the target image from a preset set of reference images, comprising:

[0026] Based on the farthest point sampling technique, multiple pose information is obtained by sampling offline pose information, and the images corresponding to the multiple pose information constitute the reference image set; the offline pose information is the pose information of the target object obtained by 3D reconstruction in an offline state;

[0027] The similarity between each reference image in the reference image set and the target image is determined respectively;

[0028] The reference image corresponding to the maximum value in the similarity score is used as a reference image similar to the target image.

[0029] Through the above implementation methods, the embodiments of this application can obtain the offline pose information of the target object through three-dimensional reconstruction in an offline state, and obtain a set of reference images from the offline pose information, which can reduce the subsequent computational load; and, by taking the reference image with the highest similarity to the target image as the similar image to the target image, accurate data can be provided for the subsequent determination of the pose information of the target object.

[0030] An optional implementation includes determining the pose information of the target object based on the pose information of the reference image, comprising:

[0031] The pose information of a reference image similar to the target image is used as the initial pose information of the target image;

[0032] From the pose information corresponding to each reference image in the reference image set, determine multiple pose information that are similar to the initial pose information;

[0033] Regression processing is performed on the query image, the reference image corresponding to the plurality of pose information, the plurality of pose information and the initial pose information to obtain the pose residual corresponding to the query image;

[0034] The initial attitude information is updated based on the attitude residual to obtain the attitude information of the target object.

[0035] By using the above implementation method, after taking the pose information of a reference image similar to the target image as the initial pose information of the target object, the pose information of the target object is obtained by updating the initial pose information based on the pose residual. This can improve the accuracy of the determined pose information.

[0036] An optional implementation further includes, after obtaining the pose information of the target object, the method further comprising:

[0037] If the query image is a frame in a video sequence, then the pose information of multiple consecutive frames in the video sequence is obtained, and the last frame in the multiple frames is the previous frame of the query image.

[0038] The pose information of the target object and the pose information of the multi-frame images are weighted and averaged to obtain the updated initial pose information of the target object;

[0039] From the pose information corresponding to each reference image in the reference image set, determine multiple pose information that are similar to the updated initial pose information;

[0040] Regression processing is performed on the query image, the reference image corresponding to the plurality of pose information, the plurality of pose information, and the updated initial pose information to obtain the pose residual corresponding to the query image;

[0041] The updated initial attitude information is updated based on the attitude residual to obtain the updated attitude information of the target object.

[0042] Through the above implementation methods, when determining that the query image is a frame in a video sequence, this application uses a temporal smoothing method to perform a weighted average of the pose information of the target object and the pose information of multiple consecutive frames before the query image to obtain updated pose information, thereby avoiding the problem of envelope cuboid jitter in the pose information generation when continuously generating videos.

[0043] An optional implementation of the method further includes:

[0044] The posture information of the target object is sent to the control device so that the control device can grasp the target object based on the posture information.

[0045] Through the above implementation methods, after determining the posture information of the target object based on the query image, the posture information of the target object can be sent to the control device so that the control device can grasp the target object based on the posture information, thus simplifying the process of unmanned grasping of the target object.

[0046] Secondly, embodiments of this application also provide an attitude estimation device, comprising:

[0047] A communication unit is used to acquire a query image, which is used to query the pose information of a target object;

[0048] An extraction unit is configured to, if the target object is an unknown target, input the query image into a large-scale model and extract the target image corresponding to the target object in the query image based on the large-scale model;

[0049] The determining unit is configured to determine a reference image similar to the target image from a set of reference images; and to determine the pose information of the target object based on the pose information of the reference image.

[0050] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the attitude estimation method of any one of the first aspects described above.

[0051] Fourthly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, the processor enables the processor to implement any of the attitude estimation methods described in the first aspect.

[0052] The technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 An application scenario diagram of an attitude estimation method provided in the embodiments of this application;

[0055] Figure 2 A hardware distribution diagram provided for an embodiment of this application;

[0056] Figure 3 An application scenario diagram for another attitude estimation method provided in the embodiments of this application;

[0057] Figure 4 A flowchart illustrating an attitude estimation method provided in an embodiment of this application;

[0058] Figure 5 A flowchart illustrating a three-dimensional reconstruction process provided in an embodiment of this application;

[0059] Figure 6 A schematic diagram illustrating a three-dimensional point cloud reconstruction effect provided in an embodiment of this application;

[0060] Figure 7 A schematic diagram illustrating a specific use case of an attitude estimation method provided in this application embodiment;

[0061] Figure 8 A schematic diagram illustrating a specific use case of another attitude estimation method provided in the embodiments of this application;

[0062] Figure 9 A complete flowchart of an attitude estimation method provided in this application embodiment is shown below;

[0063] Figure 10 This is a schematic diagram of the structure of an attitude estimation device provided in an embodiment of this application;

[0064] Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] It should be noted that the terms "comprising" and "having" and their variations used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0067] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0068] (1) Farthest Point Sampling (FPS): This technique is used to uniformly sample M points on a point cloud with N points, so that these points can better represent the overall outline of the point cloud.

[0069] (2) Structure from Motion (SFM) technique: This technique estimates three-dimensional structures from a series of two-dimensional images containing visual motion information.

[0070] (3) Large-scale models refer to artificial neural network models with a very large number of parameters. Large-scale models have strong generalization ability and good prediction ability for unknown data.

[0071] In this application, the large-scale model includes the large-scale image-text model and the large-scale object segmentation model. Specifically, the large-scale image-text model refers to the large-scale image-text model, and the large-scale object segmentation model refers to the large-scale object segmentation model.

[0072] The word “exemplary” as used below means “serving as an example, embodiment, or illustration.” Any embodiment illustrated as an “exemplary” need not be construed as superior to or better than other embodiments.

[0073] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0074] In existing technologies, traditional pose recognition methods mostly use image, point cloud information and depth information to estimate the pose of the target object to obtain pose information, which is costly and has a cumbersome processing flow.

[0075] Based on this, embodiments of this application provide a pose estimation method, apparatus, storage medium, and electronic device, which help improve the efficiency of pose estimation. The pose estimation method includes: acquiring a query image, which is used to query pose information of a target object; when the target object is unknown, inputting the query image into a large-scale model, and extracting a target image corresponding to the target object from the query image based on the large-scale model; determining a reference image similar to the target image from a set of reference images; and determining the pose information of the target object based on the pose information of the reference image.

[0076] To better understand the technical solutions provided in the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions provided in the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs.

[0077] Figure 1This illustration shows an application scenario of the attitude estimation method provided in this application. For example, this application scenario can be a scenario such as unmanned object grasping in a smart construction site. Figure 1 As shown, the application scenario includes a robot 10, an image processing device 20, and a target object 30.

[0078] The image processing device 20 can be a terminal device or a server. Data can be transmitted between the robot 10 and the image processing device 20 via a communication network.

[0079] Robot 10 collects data from its surrounding environment to obtain a query image. This query image is used to retrieve the pose information of the target object 30. Robot 10 then sends the acquired query image to image processing device 20.

[0080] Image processing device 20 extracts the target image corresponding to the target object 30 from the query image and determines a reference image similar to the target image from a pre-set set of reference images. Image processing device 20 can determine the pose information of the target object based on the pose information of the reference image. In the process of unmanned target object grasping, only the query image needs to be acquired to obtain the pose information of the target object, simplifying the pose estimation process and improving its efficiency.

[0081] like Figure 2 As shown, robot 10 may include a main unit, a robotic arm, and an image acquisition device. The image acquisition device, which can be an RGB camera, is mounted on the robotic arm. The image acquisition device can move closer to or further away from the target object as the robotic arm moves, enabling real-time monitoring of the target object's 6D pose.

[0082] Robot 10 can acquire query images in real time via an image acquisition device and send the acquired query images to a control device, which then sends the query images to image processing device 20. Image processing device 20 determines the pose information of the target object based on the received query images and sends the pose information to the robot host. The robot host then controls the robotic arm to grasp the target object based on the received pose information.

[0083] Figure 3 This illustrates another application scenario of the attitude estimation method provided in this application embodiment. For example, this application scenario could be an intelligent hoisting scenario. Figure 3 As shown, this application scenario includes a crane 10', a lifting device 20', an image acquisition device 30', and an image processing device 40'. The crane 10' can be a truck crane, crawler crane, tower crane, or rail crane, etc., and includes a boom and a hook.

[0084] The image acquisition device 30' can be an RGB camera; the image processing device 40' can be a terminal device or a server. Data can be transmitted between the crane 10' and the image processing device 40' via a communication network.

[0085] Image acquisition unit 30' acquires images of the surrounding environment to obtain a query image containing the hook. This query image is used to query the hook's attitude information. Image acquisition unit 30' then sends the query image to image processing device 40'.

[0086] Image processing device 40' extracts the target image corresponding to the hook from the query image and determines a reference image similar to the target image from a pre-set set of reference images. Image processing device 40' can determine the hook's attitude information based on the attitude information of the reference image. Image processing device 40' sends the determined hook's attitude information to crane 10'.

[0087] Based on the received hook attitude information, the crane 10′ controls itself to approach and lift the load with the correct hook attitude.

[0088] To further illustrate the technical solutions provided by the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided by the embodiments of this application. In actual processing or when the device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0089] The following is combined Figure 1 and Figure 2 The application scenarios shown are for reference. Figure 4 The flowchart shown illustrates an attitude estimation method, which explains the technical solution provided in the embodiments of this application.

[0090] S401: Obtain the query image. The query image is used to retrieve the pose information of the target object.

[0091] In some scenarios, embodiments of this application can acquire a query image captured by an image acquisition device. The image acquisition device can be a conventional RGB camera. For example, an RGB camera captures images of the surrounding environment to obtain the query image.

[0092] In other scenarios, embodiments of this application can obtain the query image from the control device. The query image can be acquired by an image acquisition device within the control device, or it can be obtained by the control device from an image acquisition device.

[0093] S402: If the target object is an unknown target, the query image is input into a large-scale model, and the target image corresponding to the target object in the query image is extracted based on the large-scale model.

[0094] In this embodiment, after obtaining the query image, it is determined whether the target object to be queried is a known target. When the target object is determined to be an unknown target, the query image can be input into a large-scale model, and target detection can be performed on the query image based on the large-scale model to determine the coordinate bounding box of the target object in the query image. Based on the determined coordinate bounding box, the target image corresponding to the target object is extracted from the query image.

[0095] S403: Determine a reference image similar to the target image from the set of reference images.

[0096] In some embodiments of this application, a set of reference images is obtained from offline pose information before determining a reference image similar to the target image.

[0097] S404: Determine the pose information of the target object based on the pose information of the reference image.

[0098] This application embodiment can determine a reference image similar to the target image from a set of reference images with known pose information, and then determine the pose information of the target object based on the pose information of the determined reference image, which simplifies the pose information estimation process and improves the pose information estimation efficiency.

[0099] exist Figure 1 In the application scenario shown, the robot needs to determine the presence of target objects within the field of view of the image acquisition unit during its movement. This embodiment of the application can determine the pose information of the target object based on the query image obtained from the robot, and then send the determined pose information to the robot.

[0100] In some embodiments, the target object appearing within the field of view of the image acquisition device can be a known target or an unknown target. The type of target object can be manually set. For example, when setting the target object to be grasped by the robot, the user can specify whether the target object is a known target or an unknown target. A known target refers to a target object that appears during the target detector training task, while an unknown target object is one that does not appear. The process of extracting the target image is described below based on these two scenarios.

[0101] Case 1: The target object is a known target.

[0102] In some embodiments, after determining that the target object for attitude estimation is a known target, the obtained query image is input into the target detector, and the target detector performs target detection on the query image to determine the coordinate box of the target object in the query image.

[0103] If the coordinate frame of the target object is determined to be non-empty in this embodiment of the application, the image region corresponding to the coordinate frame in the query image is extracted and the image region is used as the target image.

[0104] If this embodiment determines that the bounding box of the target object is empty, then the pose information of the target object is determined to be empty. In some examples, when this embodiment determines that the pose information of the target object is empty, it can inform the robot that the query image does not contain the target object and that the pose information of the target object is empty.

[0105] In order for the object detector to accurately determine the bounding box of the target object, the object detector used in the above process needs to be trained in advance. The sample data used by the object detector is the image data corresponding to the target.

[0106] Specifically, the target detector can be trained in the following ways according to the embodiments of this application:

[0107] In practice, this application embodiment acquires images containing the target object and uses these images as sample images in the training dataset. This application embodiment marks the coordinate bounding boxes of the target objects on the acquired sample images to obtain a sample dataset. The sample dataset includes multiple sample images and the coordinate bounding boxes of the target objects contained in each sample image.

[0108] In this embodiment, sample images from the sample dataset are input into the target detector. Based on the YOLOX algorithm, the coordinate boxes of the target objects output by the target detector are obtained. The network parameters of the target detector are adjusted according to the coordinate boxes of the target objects and the corresponding sample coordinate boxes until the trained target detector is obtained. The trained target detector performs ideally in scenarios such as strong light, low light, complex outdoor backgrounds, and long distances.

[0109] The training process of the target detector is described below using a hook as an example: This embodiment can acquire 667 image data points and mark the coordinate boxes of the hook on the acquired image data. The marked coordinate boxes can be divided into training coordinate boxes and test coordinate boxes. The number of training coordinate boxes can be 950, and the number of test coordinate boxes can be 92. This embodiment inputs the image data into the target detector and uses the YOLOX algorithm to train the target detector. When the mAP (maximum accuracy) on the test coordinate box reaches a set value using the YOLOX algorithm, the trained target detector is obtained. For example, when the mAP reaches 87.7%, the first target detector is considered to have completed training.

[0110] Scenario 2: The target object is an unknown target.

[0111] In some embodiments, when the target object is determined to be an unknown target, the text information corresponding to the target object is obtained. For example, if the target object is an unknown target, when obtaining the query image, the text information corresponding to the unknown target is also obtained simultaneously.

[0112] For example, when the user sets the robot to grasp an unknown target, the robot can generate text information corresponding to that target object. If the unknown target is a teacup, the text information corresponding to the teacup could be "This is a picture containing a teacup."

[0113] In this embodiment, text information and query images can be input into a large-scale model, and target detection can be performed on the query images based on the large-scale model to determine the coordinate boxes of target objects in the query images.

[0114] In some embodiments, the large-scale model may include a large-scale image-text model and a large-scale object segmentation model. The large-scale image-text model can receive input of image and text pairs and return feature regions of the queried target object. The large-scale object segmentation model can accept prompts such as points, bounding boxes, and masks to segment the target object.

[0115] In this embodiment, text information and query images are input into a large-scale image model within a large-scale model. Based on the large-scale image-text model, feature extraction is performed on the query image to obtain the feature region of the target object.

[0116] In some embodiments, if the threshold of the feature region output by the large-scale image-text model is higher than a set threshold, the feature region is input into the large-scale object segmentation model. If the threshold of the feature region output by the large-scale image-text model is lower than the set threshold, the bounding box of the target object is determined to be empty, and the pose information of the target object is determined to be empty.

[0117] In this embodiment, the feature region of the target object is input into the large-scale object segmentation model in the large-scale model, and the feature region is cut based on the large-scale object segmentation model to obtain the coordinate frame of the target object.

[0118] In some embodiments, this application can select the largest feature region from the feature regions output by a large-scale image and text model and input it into a large-scale object segmentation model. Furthermore, this application can also embed the point with the highest confidence level within the largest feature region, the minimum bounding rectangle of the largest feature region, and the mask of the largest feature region as prompt words and input them into the large-scale object segmentation model to obtain an accurate target segmentation mask, and take the minimum bounding rectangle as the coordinate frame of the target object. Wherein, when the feature region mask has holes, the holes are filled.

[0119] For example, embodiments of this application can utilize the feature extraction capabilities of large-scale image and text models to determine the approximate location of a target object, and then use a large-scale object segmentation model to precisely segment the approximate location of the target object to obtain the precise coordinate frame of the target object.

[0120] In this embodiment of the application, after obtaining the coordinate frame of the target object, the image region corresponding to the coordinate frame in the query image is extracted, and the image region is used as the target image.

[0121] In some embodiments, when extracting the image region corresponding to the coordinate frame, to avoid truncating the target object, the embodiments of this application may further enlarge the coordinate frame, then perform center cropping on the target object, and scale the target object according to the size of (128, 128) to obtain the target image. For example, the enlargement ratio of the coordinate frame can be a length-to-width ratio of 1.1:1.

[0122] After obtaining the target image, this embodiment of the application can further sample the offline pose information using the farthest point sampling technique to obtain multiple pose information sets. The images corresponding to the multiple pose information sets constitute a reference image set. The offline pose information refers to the pose information of the target object obtained through 3D reconstruction in an offline state, as described in this embodiment of the application.

[0123] In some embodiments, the offline attitude information can be generated through the following steps:

[0124] This application's embodiments acquire video sequences containing target objects. The target objects include known targets and unknown targets.

[0125] In some embodiments, the video sequence is acquired in a single-background experimental setting, and the duration of the acquired video sequence is controlled to be approximately one minute. Furthermore, the video sequence acquired in this embodiment needs to include various perspectives of the target object, as well as capturing images of the target object from a distance to a closer viewpoint within the same perspective before switching perspectives.

[0126] In this embodiment of the application, after obtaining the video sequence, SFM technology can be used to perform three-dimensional reconstruction of the video sequence to obtain the offline pose information of the target object in the video sequence.

[0127] Taking the Colmap algorithm as an example, the process of 3D reconstruction using SFM technology is explained: Figure 5 As shown, in this embodiment, the width, height, and name of each frame in the video sequence are written to database.db. For each frame, feature extraction, feature matching, incremental reconstruction, image distortion correction, and depth estimation are performed step-by-step to obtain the 3D point cloud reconstruction effect and the camera's intrinsic and extrinsic parameters and the offline pose information of the target object for each frame. For example, when the target object is a hook, the 3D point cloud reconstruction effect obtained by this embodiment is as follows: Figure 6 As shown.

[0128] Considering the limited edge computing resources in construction site scenarios, in order to cover the pose information of target objects from various perspectives as much as possible during the pose estimation process, the Farthest Point Sampling (FPS) technique can be used to downsample the obtained offline pose information.

[0129] The embodiments of this application can use the farthest point sampling technique to downsample the offline pose information of the target object in the video sequence to obtain multiple pose information, and the images corresponding to the multiple pose information form a reference image set.

[0130] In some embodiments, the offline attitude information can be downsampled using the following formula:

[0131]

[0132] The initial sample for offline attitude information is set as follows: Sampling points N, {s i} represents the offline pose information of the target object from the perspective of each frame; d(q,S) is the farthest distance from the currently selected point to set S, and d(p,S) is the farthest distance from the next point to set S. Distances can be Euclidean distance, absolute distance, etc.

[0133] This application embodiment utilizes FPS technology to downsample the pose information of the offline target object in each frame of the obtained video sequence to the pose information of N reference images. Furthermore, by employing FPS technology, the amount of original data information can be preserved to the maximum extent, significantly reducing computational resources and the time consumed in subsequent matching and refinement, thereby reducing the time required for a single inference operation to complete pose estimation.

[0134] In some embodiments, this application can further perform farthest-point sampling processing on the offline pose information during the offline stage to obtain a reference image set. This application can also directly obtain the reference image set obtained in the offline stage during the pose estimation stage.

[0135] In this embodiment of the application, after obtaining the set of reference images, each reference image can be rotated in the plane by directional angles of -90°, -45°, 0°, 45°, and 90° to obtain more reference images that match the query images.

[0136] In this embodiment of the application, after obtaining a set of reference images, a reference image similar to the target image can be determined from the set of reference images.

[0137] In some embodiments, the similarity between each reference image in the reference image set and the target image can be determined, and the reference image corresponding to the maximum similarity can be used as the reference image similar to the target image.

[0138] In some examples, embodiments of this application may input the target image obtained by an object detector or a large-scale model into the selector. Embodiments of this application may also obtain an offline-generated set of reference images and input this set into the selector. Embodiments of this application may use a neural network in the selector to calculate the similarity between the target image and the reference images in the set, thereby obtaining the reference image most similar to the target image.

[0139] In this embodiment of the application, after obtaining a reference image similar to the target image, the pose information of the target object is determined based on the pose information of the reference image.

[0140] In some embodiments, the pose information of a reference image similar to the target image can be used as the initial pose information of the target image. Alternatively, the obtained initial pose information can be input into a refiner to converge the initial pose information and obtain the pose information of the target object.

[0141] In this embodiment of the application, after obtaining the initial pose information, multiple pose information similar to the initial pose information are determined from the pose information corresponding to each reference image in the reference image set.

[0142] The embodiments of this application can perform regression processing on the query image, the reference image corresponding to multiple pose information, the multiple pose information and the initial pose information to obtain the pose residual corresponding to the query image, and update the initial pose information according to the pose residual to obtain the pose information of the target object.

[0143] In some embodiments, the present application embodiments can input the reference image corresponding to the obtained multiple pose information, the multiple pose information, the query image, and the initial pose information into the refiner to regress the pose residual in this state. The refiner can be a 3D convolutional neural network (CNN).

[0144] In this embodiment, a reference image, multiple pose information, a query image, and initial pose information corresponding to multiple pose information can be input into a 3D CNN. Based on the 3D CNN, the inconsistencies between the reference image, multiple pose information, the query image, and the initial pose information corresponding to multiple pose information are analyzed, and the pose residuals are obtained by regression.

[0145] This application embodiment can update the initial attitude information based on the attitude residual to obtain the attitude information of the target object. For example, this application embodiment can also set the iteration parameter in the refiner, for example, the iteration parameter can be set to 3. As another example, this application embodiment can also iteratively update the initial attitude information based on the attitude residual until the obtained attitude residual is less than a set threshold, and then output the attitude information of the target object.

[0146] For example, in this embodiment of the application, the initial pose information, the six pose information most similar to the initial pose information in the pose information corresponding to the reference image set, the reference image corresponding to the six pose information and the query image are input into the refiner, the pose residual in this state is regressed, the pose residual is updated to update the initial pose information, and then three iterations are performed to output the pose information of the target object.

[0147] In some embodiments, when the query image is a single image, the present application embodiments can send the above-obtained posture information to the control device so that the control device can grasp the target object based on the posture information.

[0148] The control device can be a robot, the main robot unit within a robot, or a crane.

[0149] In some embodiments, when the query image is a frame in a video sequence, this application may also employ a temporal smoothing method to address the problem of envelope cuboid jitter in the generation of pose information when continuously generating videos.

[0150] After obtaining the pose information of the target object, this embodiment of the application can also obtain the pose information of multiple consecutive frames of images from the video sequence corresponding to the query image. The last frame of the multiple frames is the previous frame of the query image.

[0151] For example, embodiments of this application can obtain the pose information of five consecutive frames of images in a video sequence that precede the query image.

[0152] The embodiments of this application can also perform a weighted average of the pose information of the target object and the pose information of multiple frames of images to obtain the updated initial pose information of the target object.

[0153] In some embodiments, the updated initial attitude information can be obtained through the following methods:

[0154] Pr` t =aPr t +bPr t-1 +cPr t-2 +dPr t-3 +ePr t-4

[0155] Where a, b, c, d, and e are weight coefficients for different consecutive frames, which are summed to 1; t represents the frame number of the current frame; Pr` t This indicates the updated initial pose information of the queried image; Pr t This indicates the pose information of the target object in the query image; Pr t-1 This indicates the pose information of the target object in the previous frame of the query image; Pr t-2 This indicates the pose information of the target object in the previous two frames of the query image; Pr t-3 This indicates the pose information of the target object in the previous 3 frames of the query image; Pr t-4 This indicates the pose information of the target object in the previous four frames of the query image.

[0156] In this embodiment, after obtaining the updated initial pose information, multiple pose information similar to the updated initial pose information are determined from the pose information corresponding to each reference image in the reference image set. This embodiment can also perform regression processing on the multiple pose information, the reference images corresponding to the multiple pose information, the query image, and the updated initial pose information to obtain the pose residual of the query image, and update the obtained updated initial pose information based on the obtained pose residual to obtain the updated pose information of the target object.

[0157] For example, in this embodiment of the application, the updated initial pose information, multiple pose information similar to the updated initial pose information, reference images corresponding to the multiple pose information, and query images can also be input into the refiner to obtain the updated pose information of the target object. This process is the same as the process described above of inputting the initial pose information, multiple pose information similar to the initial pose information, reference images corresponding to the multiple pose information, and query images into the refiner to obtain the pose information of the target object, and will not be repeated here.

[0158] In some embodiments, this application provides a specific use case of an attitude estimation method. For example... Figure 7 As shown in the embodiment of this application, after obtaining the query image, the query image is input to the target detector. The query image is used to query the pose information of the target object, which is a known target.

[0159] This application embodiment uses a target detector to perform target detection on the query image to determine whether there is a target object in the query image.

[0160] In some embodiments, when a target object is found in the query image, the target image corresponding to the target object in the query image is input into the matcher. Embodiments of this application can also improve the matcher by sampling offline pose information to obtain a set of reference images. Embodiments of this application use the matcher to select the reference image most similar to the target image from the set of reference images, and use the pose information of this reference image as the initial pose information of the target image.

[0161] In this embodiment, the initial pose information, the query image, and the pose information corresponding to the reference image set, along with multiple pose information pieces similar to the initial pose information and the reference images corresponding to these pose information pieces, can be input into the refiner. The refiner performs regression processing to obtain the pose information of the target object. For example, the refiner can be a 3D CNN.

[0162] After obtaining the pose information of the target object, this embodiment outputs the pose information of the target object. In some embodiments, when there is no target object in the query image, this embodiment determines that the pose information of the target object is empty and outputs the pose information.

[0163] In some embodiments, this application provides specific use cases of another attitude estimation method. For example... Figure 8 As shown in the embodiment of this application, after obtaining the query image, the target object to be queried is determined to be an unknown target, and the text information corresponding to the target object is obtained. This embodiment of the application inputs the query image and text information into a large-scale image-text model within a large-scale model, and performs target detection on the query image based on the large-scale image-text model.

[0164] When no target object is found in the query image, this embodiment of the application determines that the pose information of the target object is empty and outputs the pose information.

[0165] When a target object is found in the query image, this embodiment outputs the feature region of the target object through a large-scale image-text model, and inputs the feature region into a large-scale object segmentation model. Based on the large-scale object segmentation model, the feature region is segmented to obtain the coordinate bounding box of the target object. This embodiment extracts the target image of the target object based on the coordinate bounding box, and inputs the target image into a matcher to determine the initial pose information of the target image. This process is similar to... Figure 7 The process of determining the initial attitude information through the matcher is the same, and will not be repeated here.

[0166] This embodiment of the application inputs initial posture information, multiple posture information pieces similar to the initial posture information from the posture information corresponding to the reference image set, the reference image corresponding to the multiple posture information pieces, and the query image into the refiner to obtain the posture information of the target object. This process is similar to... Figure 7 The process of obtaining the target object's posture information through the refiner is the same, and will not be repeated here.

[0167] based on Figure 8 The present application embodiment describes a method that inputs a query image and the corresponding text information of the target object into a large-scale image-text model. Based on this model, the unknown target is detected, yielding its target image. The pose information of the unknown target can be obtained without training on the unknown image; therefore, the pose estimation method provided in this application embodiment has extremely strong generalization capabilities. Furthermore, this application can also determine the pose information of the target object based on the pose information of an offline reference image, thereby reducing the complexity of the pose estimation process and improving estimation efficiency.

[0168] like Figure 9 As shown in the diagram, this application provides a complete flowchart of an attitude estimation method, taking the target object as a known target as an example, including the following steps:

[0169] S901: Obtain the query image. The query image is used to retrieve the pose information of the target object.

[0170] S902: Input the query image into the target detector, perform target detection on the query image based on the target detector, and determine the coordinate box of the target object in the query image.

[0171] In some embodiments, when determining the coordinate frame of the target object, this application embodiment also needs to determine whether the coordinate frame is empty. If the coordinate frame is empty, the pose information of the target object is determined to be empty. If the coordinate frame is not empty, step S903 is executed.

[0172] S903: Extract the image region corresponding to the coordinate box in the query image and use the image region as the target image.

[0173] S904: Determine the similarity between each reference image in the reference image set and the target image.

[0174] In some embodiments, before determining the similarity between each reference image in the reference image set and the target image, this application embodiment may further sample the offline pose information based on the farthest point sampling technique to obtain multiple pose information, and the images corresponding to the multiple pose information constitute the reference image set. The offline pose information is the pose information of the target object obtained through 3D reconstruction in an offline state.

[0175] S905: Use the reference image corresponding to the maximum similarity value as the reference image that is similar to the target image.

[0176] S906: Use the pose information of a reference image similar to the target image as the initial pose information of the target image.

[0177] S907: From the pose information corresponding to each reference image in the reference image set, determine multiple pose information that are similar to the initial pose information.

[0178] S908: Perform regression processing on the query image, the reference image corresponding to multiple pose information, the multiple pose information and the initial pose information to obtain the pose residual corresponding to the query image.

[0179] S909: Update the initial attitude information based on the attitude residual to obtain the attitude information of the target object.

[0180] S910: Determine whether the query image is a single image; if yes, proceed to step S911; if no, proceed to step S912.

[0181] S911: Send the orientation information of the target object to the control device so that the control device can grasp the target object based on the orientation information of the target object.

[0182] S912: Obtain pose information of multiple consecutive frames of images from the video sequence corresponding to the query image.

[0183] In this context, the last frame of a multi-frame image is the previous frame of the queried image.

[0184] S913: Perform a weighted average of the pose information of the target object and the pose information of multiple frames to obtain the updated initial pose information of the target object.

[0185] S914: From the pose information corresponding to each reference image in the reference image set, determine multiple pose information that are similar to the updated initial pose information.

[0186] S915: Perform regression processing on the query image, the reference image corresponding to multiple pose information, the multiple pose information, and the updated initial pose information to obtain the pose residual corresponding to the query image.

[0187] S916: Based on the updated initial attitude information obtained from the attitude residual update, the updated attitude information of the target object is obtained.

[0188] S917: Send the updated attitude information to the control device so that the control device can grasp the target object based on the updated attitude information.

[0189] based on Figure 9 The embodiments of this application can determine the pose information of a target object based on the pose information of reference images similar to the target image in a set of reference images. This eliminates the need for online 3D reconstruction, simplifies the pose estimation process, improves the efficiency of pose estimation, and ensures the accuracy of pose estimation. Furthermore, the embodiments of this application can also perform pose estimation for both known and unknown targets, improving generalization.

[0190] Based on the same inventive concept, this application also provides an attitude estimation device. Since the principle of this device in solving the problem is similar to the attitude estimation method described above, this device can be implemented with reference to the above method embodiments, and repeated details will not be described again.

[0191] like Figure 10 As shown in the figure, a structural block diagram of an attitude estimation device provided in this application embodiment is provided. The device includes:

[0192] Communication unit 1001 is used to acquire a query image, the query image being used to query the pose information of a target object;

[0193] Extraction unit 1002 is used to input the query image into a large-scale model if the target object is an unknown target, and extract the target image corresponding to the target object in the query image based on the large-scale model;

[0194] The determining unit 1003 is used to determine a reference image similar to the target image from the set of reference images; and to determine the pose information of the target object based on the pose information of the reference image.

[0195] In one optional implementation, the extraction unit 1002 is specifically used for:

[0196] Obtain the text information corresponding to the target object;

[0197] The text information and the query image are input into the large-scale model, and the target detection is performed on the query image based on the large-scale model to determine the coordinate bounding box of the target object in the query image;

[0198] Extract the image region corresponding to the coordinate frame in the query image, and use the image region as the target image.

[0199] An optional implementation is that the large-scale model includes a large-scale image and text model and a large-scale object segmentation model; the extraction unit 1002 is specifically used for:

[0200] The text information and the query image are input into the large-scale image-text model, and feature extraction is performed on the query image based on the large-scale image-text model to determine the feature region of the target object;

[0201] The feature region is input into the large-scale object segmentation model, and the feature region is segmented based on the large-scale object segmentation model to obtain the coordinate box of the target object.

[0202] In one optional implementation, the extraction unit 1002 is further configured to:

[0203] If the target object is a known target, the query image is input into the target detector, and the target detection is performed on the query image based on the target detector to determine the coordinate bounding box of the target object in the query image;

[0204] Extract the image region corresponding to the coordinate frame in the query image, and use the image region as the target image.

[0205] In one optional implementation, the determining unit 1003 is specifically used for:

[0206] Based on the farthest point sampling technique, multiple pose information is obtained by sampling offline pose information, and the images corresponding to the multiple pose information constitute the reference image set; the offline pose information is the pose information of the target object obtained by 3D reconstruction in an offline state;

[0207] The similarity between each reference image in the reference image set and the target image is determined respectively;

[0208] The reference image corresponding to the maximum value in the similarity score is used as a reference image similar to the target image.

[0209] In one optional implementation, the determining unit 1003 is specifically used for:

[0210] The pose information of a reference image similar to the target image is used as the initial pose information of the target image;

[0211] From the pose information corresponding to each reference image in the reference image set, determine multiple pose information that are similar to the initial pose information;

[0212] Regression processing is performed on the query image, the reference image corresponding to the plurality of pose information, the plurality of pose information and the initial pose information to obtain the pose residual corresponding to the query image;

[0213] The initial attitude information is updated based on the attitude residual to obtain the attitude information of the target object.

[0214] In one optional implementation, after obtaining the pose information of the target object, the determining unit 1003 is further configured to:

[0215] If the query image is a frame in a video sequence, then the pose information of multiple consecutive frames in the video sequence is obtained, and the last frame in the multiple frames is the previous frame of the query image.

[0216] The pose information of the target object and the pose information of the multi-frame images are weighted and averaged to obtain the updated initial pose information of the target object;

[0217] From the pose information corresponding to each reference image in the reference image set, determine multiple pose information that are similar to the updated initial pose information;

[0218] Regression processing is performed on the query image, the reference image corresponding to the plurality of pose information, the plurality of pose information, and the updated initial pose information to obtain the pose residual corresponding to the query image;

[0219] The updated initial attitude information is updated based on the attitude residual to obtain the updated attitude information of the target object.

[0220] In one optional implementation, the communication unit 1001 is further configured to:

[0221] The posture information of the target object is sent to the control device so that the control device can grasp the target object based on the posture information.

[0222] Corresponding to the embodiments of the attitude estimation method described above, this application also provides an electronic device. This electronic device can be a server, a mobile terminal, or a computer, etc. For example, this electronic device can be an industrial control device.

[0223] The electronic device includes at least a memory for storing data and a processor for data processing. The processor for data processing can be implemented using a microprocessor, CPU, GPU (Graphics Processing Unit), DSP, or FPGA. The memory stores operation instructions, which can be computer-executable code, to implement the various steps in the attitude estimation method described in the embodiments of this application.

[0224] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 11 As shown, in this embodiment of the application, the electronic device 100 includes: a processor 101, a display 102, a memory 103, an input device 106, a bus 105, and a communication module 104; the processor 101, memory 103, input device 106, display 102, and communication module 104 are all connected through the bus 105, which is used for data transmission between the processor 101, memory 103, display 102, communication module 104, and input device 106.

[0225] The memory 103 can be used to store software programs and modules, such as the program instructions / modules corresponding to the attitude estimation method in this embodiment. The processor 101 executes various functional applications and data processing of the electronic device 100 by running the software programs and modules stored in the memory 103, such as the attitude estimation method provided in this embodiment. The memory 103 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs of at least one application, etc.; the data storage area may store data created according to the use of the electronic device 100 (such as reference image sets and other related data). In addition, the memory 103 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0226] The processor 101 is the control center of the electronic device 100. It connects various parts of the electronic device 100 via the bus 105 and various interfaces and lines. It performs various functions of the electronic device 100 and processes data by running or executing software programs and / or modules stored in the memory 103 and calling data stored in the memory 103. Optionally, the processor 101 may include one or more processing units, such as a CPU, GPU, digital processing unit, etc.

[0227] The processor 101 can display the attitude estimation results to the user via the display 102.

[0228] The processor 101 can also connect to the network via the communication module 104 to obtain query images, etc.

[0229] Input device 106 is mainly used to obtain user input operations, and the input device 106 may also be different depending on the electronic device. For example, when the electronic device is a computer, the input device 106 can be an input device such as a mouse or keyboard; when the electronic device is a smartphone, tablet computer, or other portable device, the input device 106 can be a touch screen.

[0230] This application also provides a computing device readable storage medium for attitude estimation methods, meaning that the content is not lost after power failure. This storage medium stores software programs, including program code. When the program code runs on a computing device, the software program, when read and executed by one or more processors, can implement any of the attitude estimation methods described in this application.

[0231] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0232] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0233] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0234] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0235] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A pose estimation method, characterized in that, The method includes: Obtain a query image, which is used to query the pose information of a target object; If the target object is an unknown target, then obtain the text information corresponding to the target object; input the text information and the query image into a large-scale image-text model, extract features from the query image based on the large-scale image-text model, and select the largest feature region from the feature regions output by the large-scale image-text model as the feature region of the target object; input the largest feature region, the point with the highest built-in confidence of the largest feature region, the minimum bounding rectangle of the largest feature region, and the mask of the largest feature region into a large-scale object segmentation model, and segment the feature region based on the large-scale object segmentation model to obtain the coordinate frame of the target object; extract the image region corresponding to the coordinate frame in the query image, and use the image region as the target image corresponding to the target object; if the threshold of the feature region output by the large-scale image-text model is higher than a set threshold, then input the feature region into the large-scale object segmentation model; if the threshold of the feature region output by the large-scale image-text model is lower than the set threshold, then determine that the coordinate frame of the target object is empty, and the pose information of the target object is empty; Determine reference images similar to the target image from the set of reference images; The pose information of the target object is determined based on the pose information of the reference image.

2. The method according to claim 1, characterized in that, The method further includes: If the target object is a known target, the query image is input into the target detector, and the target detection is performed on the query image based on the target detector to determine the coordinate bounding box of the target object in the query image; Extract the image region corresponding to the coordinate frame in the query image, and use the image region as the target image.

3. The method according to claim 1 or 2, characterized in that, The step of determining a reference image similar to the target image from the set of reference images includes: Based on the farthest point sampling technique, multiple pose information is obtained by sampling offline pose information, and the images corresponding to the multiple pose information constitute the reference image set; the offline pose information is the pose information of the target object obtained by 3D reconstruction in an offline state; The similarity between each reference image in the reference image set and the target image is determined respectively; The reference image corresponding to the maximum value in the similarity score is used as a reference image similar to the target image.

4. The method according to claim 3, characterized in that, Determining the pose information of the target object based on the pose information of the reference image includes: The pose information of a reference image similar to the target image is used as the initial pose information of the target image; From the pose information corresponding to each reference image in the reference image set, determine multiple pose information that are similar to the initial pose information; Regression processing is performed on the query image, the reference image corresponding to the plurality of pose information, the plurality of pose information and the initial pose information to obtain the pose residual corresponding to the query image; The initial attitude information is updated based on the attitude residual to obtain the attitude information of the target object.

5. The method according to claim 4, characterized in that, After obtaining the pose information of the target object, the method further includes: If the query image is a frame in a video sequence, then the pose information of multiple consecutive frames in the video sequence is obtained, and the last frame in the multiple frames is the previous frame of the query image. The pose information of the target object and the pose information of the multi-frame images are weighted and averaged to obtain the updated initial pose information of the target object; From the pose information corresponding to each reference image in the reference image set, determine multiple pose information that are similar to the updated initial pose information; Regression processing is performed on the query image, the reference image corresponding to the plurality of pose information, the plurality of pose information, and the updated initial pose information to obtain the pose residual corresponding to the query image; The updated initial attitude information is updated based on the attitude residual to obtain the updated attitude information of the target object.

6. The method according to claim 4, characterized in that, The method further includes: The posture information of the target object is sent to the control device so that the control device can grasp the target object based on the posture information.

7. An attitude estimation device, characterized in that, include: A communication unit is used to acquire a query image, which is used to query the pose information of a target object; An extraction unit is configured to: if the target object is an unknown target, acquire the text information corresponding to the target object; input the text information and the query image into a large-scale image-text model; extract features from the query image based on the large-scale image-text model; select the largest feature region from the feature regions output by the large-scale image-text model as the feature region of the target object; input the largest feature region, the point with the highest built-in confidence of the largest feature region, the minimum bounding rectangle of the largest feature region, and the mask of the largest feature region into a large-scale object segmentation model; segment the feature region based on the large-scale object segmentation model to obtain the coordinate frame of the target object; extract the image region corresponding to the coordinate frame in the query image, and use the image region as the target image corresponding to the target object; A determining unit is configured to determine, from a set of reference images, a reference image similar to the target image; Based on the pose information of the reference image, determine the pose information of the target object; The step of determining a reference image similar to the target image from the set of reference images includes: Based on the farthest point sampling technique, multiple pose information is obtained by sampling offline pose information, and the images corresponding to the multiple pose information constitute the reference image set; the offline pose information is the pose information of the target object obtained by 3D reconstruction in an offline state; The similarity between each reference image in the reference image set and the target image is determined respectively; The reference image corresponding to the maximum value in the similarity score is used as a reference image similar to the target image.

8. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images