Pose estimation method and apparatus, storage medium, and electronic device

Abstract

The present application relates to the field of image processing, and provides a pose estimation method and apparatus, a storage medium, and an electronic device. The method comprises: acquiring a query image; if a target object is an unknown object, inputting the query image into a large-scale model, and on the basis of the large-scale model, extracting a target image in the query image corresponding to the target object; determining reference images similar to the target image from a reference image set; and determining pose information of the target object on the basis of pose information of the reference images. According to the present application, point cloud information and depth information are not needed in the process of performing pose estimation on the target object in the query image, so that the costs can be reduced, and the pose estimation efficiency is improved. Moreover, offline and online processes are combined in the method of the present application, and the online process is simple and clear, so that the real-time pose estimation capability is provided. According to the present application, large-scale model technology is used, and generalization is very high, so that pose estimation can be performed on a known object and an unknown object.

Description

A method, device, storage medium and electronic device for posture estimation

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on July 21, 2023, with application number 202310900683.5 and invention name “A posture estimation method, device, storage medium and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of image processing, and in particular to a posture estimation method, device, storage medium and electronic device. Background Art

[0004] With the development of intelligent technology, smart construction and unmanned operations have become current research hotspots. Currently, for three-dimensional grasping operations such as crane hook loading and unloading and agricultural machinery picking fruits, vision-based 6D posture recognition of objects such as hooks is key to achieving unmanned grasping.

[0005] At present, most methods of posture recognition use images, point cloud information and depth information to estimate the posture of the target and obtain posture information, which is costly and has a cumbersome processing flow.

[0006] Summary of the Invention

[0007] To solve existing technical problems, embodiments of the present application provide a posture estimation method, device, storage medium, and electronic device to improve the efficiency of posture estimation.

[0008] To achieve the above objectives, the technical solution of the embodiment of the present application is implemented as follows:

[0009] In a first aspect, an embodiment of the present application provides a posture estimation method, the method comprising:

[0010] Acquire a query image, where the query image is used to query posture information of a target object;

[0011] If the target object is an unknown target, inputting the query image into a large-scale model, and extracting a target image corresponding to the target object in the query image based on the large-scale model;

[0012] Determining a reference image similar to the target image from a reference image set;

[0013] The posture information of the target object is determined according to the posture information of the reference image.

[0014] In this method, after extracting the target image corresponding to the target object from the query image, the embodiment of the present application determines a reference image similar to the target image from a reference image set, and determines the pose information of the target object based on the pose information of the determined reference image. In this way, in the process of determining the pose information, the point cloud information or depth information of the target object is not required, thereby reducing costs and improving the generalization and efficiency of pose estimation. In addition, the present application adopts large-scale model technology with strong generalization, which can perform pose estimation on both known and unknown targets.

[0015] In an optional implementation, inputting the query image into a large-scale model, and extracting a target image corresponding to the target object in the query image based on the large-scale model, includes:

[0016] Obtaining text information corresponding to the target object;

[0017] Inputting the text information and the query image into the large-scale model, performing target detection on the query image based on the large-scale model, and determining a coordinate frame of the target object in the query image;

[0018] An image region corresponding to the coordinate frame in the query image is extracted, and the image region is used as the target image.

[0019] Through the above implementation, after determining that the target object is an unknown target, the embodiment of the present application obtains the text information corresponding to the target, and based on the text information, uses large-scale model technology to perform target detection on the query image, thereby realizing the detection of unknown targets and further improving the generalization of posture estimation.

[0020] In an optional implementation, the large-scale model includes a large-scale image-text model and a large-scale object segmentation model; inputting the text information and the query image into the large-scale model, performing target detection on the query image based on the large-scale model, and determining the coordinate frame of the target object in the query image includes:

[0021] Inputting the text information and the query image into the large-scale image-text model, performing feature extraction on the query image based on the large-scale image-text model, and determining a feature region of the target object;

[0022] The feature region is input into the large-scale all-things segmentation model, and the feature region is cut based on the large-scale all-things segmentation model to obtain a coordinate frame of the target object.

[0023] Through the above implementation, the embodiment of the present application can input text information and query images into a large-scale graphic model, extract features of the query image, determine the approximate location of the target object, and then perform precise cutting through a large-scale all-things segmentation model to obtain the precise coordinate frame of the target object.

[0024] In an optional implementation manner, the method further comprises:

[0025] If the target object is a known target, inputting the query image into a target detector, performing target detection on the query image based on the target detector, and determining a coordinate frame of the target object in the query image;

[0026] An image region corresponding to the coordinate frame in the query image is extracted, and the image region is used as the target image.

[0027] Through the above implementation, after determining that the target is a known target, the embodiment of the present application performs target detection on the query image through the first target detector to obtain the target image, which can improve the accuracy and efficiency of target detection.

[0028] In an optional implementation manner, determining a reference image similar to the target image from a preset reference image set includes:

[0029] Based on the farthest point sampling technology, the offline posture information is sampled to obtain multiple posture information, and the images corresponding to the multiple posture information constitute the reference image set; the offline posture information is the posture information of the target object obtained by three-dimensional reconstruction in an offline state;

[0030] respectively determining a similarity between each reference image in the reference image set and the target image;

[0031] The reference image corresponding to the maximum value of the similarities is used as a reference image similar to the target image.

[0032] Through the above-mentioned implementation mode, the embodiment of the present application can obtain the offline posture information of the target object through three-dimensional reconstruction in an offline state, and obtain a reference image set from the offline posture information, which can reduce the subsequent calculation amount; and, using the reference image with the highest similarity to the target image as the similar image to the target image, it can provide accurate data for the subsequent determination of the posture information of the target object.

[0033] In an optional implementation manner, determining the posture information of the target object based on the posture information of the reference image includes:

[0034] Using the pose information of a reference image similar to the target image as the initial pose information of the target image;

[0035] Determining, from the posture information corresponding to each reference image in the reference image set, a plurality of posture information similar to the initial posture information;

[0036] Performing regression processing on the query image, the reference images corresponding to the multiple pose information, the multiple pose information, and the initial pose information to obtain a pose residual corresponding to the query image;

[0037] The initial posture information is updated according to the posture residual to obtain the posture information of the target object.

[0038] Through the above implementation, after the posture information of the reference image similar to the target image is used as the initial posture information of the target object, the initial posture information is updated based on the posture residual to obtain the posture information of the target object, which can improve the accuracy of the determined posture information.

[0039] In an optional implementation manner, after obtaining the posture information of the target object, the method further includes:

[0040] If the query image is a frame image in a video sequence, then obtaining posture information of multiple consecutive frames of images from the video sequence, wherein the last frame image in the multiple frames of images is the previous frame image of the query image;

[0041] Performing weighted averaging on the posture information of the target object and the posture information of the multiple frames of images to obtain updated initial posture information of the target object;

[0042] Determining, from the posture information corresponding to each reference image in the reference image set, a plurality of posture information similar to the updated initial posture information;

[0043] Performing regression processing on the query image, the reference images corresponding to the multiple pose information, the multiple pose information, and the updated initial pose information to obtain a pose residual corresponding to the query image;

[0044] The updated initial posture information is updated according to the posture residual to obtain updated posture information of the target object.

[0045] Through the above-mentioned implementation, when determining that the query image is a frame image in a video sequence, the embodiment of the present application adopts a time smoothing method to perform weighted averaging on the posture information of the target object and the posture information of multiple consecutive frames of images before the query image to obtain updated posture information, so as to avoid the problem of jitter of the envelope rectangle generated by the posture information when continuously generating videos.

[0046] In an optional implementation manner, the method further comprises:

[0047] The posture information of the target object is sent to a control device, so that the control device grasps the target object according to the posture information.

[0048] Through the above implementation, after determining the posture information of the target object based on the query image, the embodiment of the present application can send the posture information of the target object to the control device, so that the control device can grasp the target object based on the posture information, simplifying the process of unmanned grasping of the target object.

[0049] In a second aspect, an embodiment of the present application further provides a posture estimation device, comprising:

[0050] a communication unit, configured to obtain a query image, wherein the query image is used to query posture information of a target object;

[0051] an extraction unit, configured to, if the target object is an unknown target, input the query image into a large-scale model, and extract a target image corresponding to the target object in the query image based on the large-scale model;

[0052] The determining unit is configured to determine a reference image similar to the target image from a reference image set; and determine the posture information of the target object based on the posture information of the reference image.

[0053] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the posture estimation method of any one of the above-mentioned first aspects is implemented.

[0054] In a fourth aspect, an embodiment of the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the processor implements any one of the posture estimation methods of the first aspect above.

[0055] The technical effects brought about by any implementation method in the second to fourth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] FIG1 is a diagram illustrating an application scenario of a posture estimation method provided by an embodiment of the present application;

[0058] FIG2 is a schematic diagram of a hardware distribution provided in an embodiment of the present application;

[0059] FIG3 is an application scenario diagram of another posture estimation method provided by an embodiment of the present application;

[0060] FIG4 is a flow chart of a posture estimation method provided in an embodiment of the present application;

[0061] FIG5 is a schematic diagram of a flow chart of a three-dimensional reconstruction process provided in an embodiment of the present application;

[0062] FIG6 is a schematic diagram of a three-dimensional point cloud reconstruction effect provided by an embodiment of the present application;

[0063] FIG7 is a schematic diagram of a specific use case of a posture estimation method provided in an embodiment of the present application;

[0064] FIG8 is a schematic diagram of a specific use case of another posture estimation method provided in an embodiment of the present application;

[0065] FIG9 is a schematic diagram of a complete flow chart of a posture estimation method provided in an embodiment of the present application;

[0066] FIG10 is a schematic structural diagram of a posture estimation device provided in an embodiment of the present application;

[0067] FIG11 is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0069] It should be noted that the terms "including" and "having" and their variations involved in the documents of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0070] The following explains some of the terms used in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0071] (1) Farthest Point Sampling (FPS): It is used to uniformly sample M points on a point cloud of N points so that these points can better represent the overall contour of the point cloud.

[0072] (2) Structure from Motion (SFM) technology: It is a technology for estimating three-dimensional structure from a series of two-dimensional image sequences containing visual motion information.

[0073] (3) Large-scale models refer to artificial neural network models with a very large number of parameters. Large-scale models have strong generalization capabilities and good predictive capabilities for unknown data.

[0074] In this application, large-scale models include large-scale Mo-image-text models and large-scale everything segmentation models. Among them, large-scale image-text models refer to large image-text models, and large-scale everything segmentation models refer to large everything segmentation models.

[0075] The word “exemplary” is used hereinafter to mean “serving as an example, example, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0076] The terms "first" and "second" are used for descriptive purposes only and should not be construed as explicitly or implicitly indicating relative importance or the number of the technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0077] In the prior art, traditional posture recognition methods mostly use images, point cloud information, and depth information to estimate the posture of the target object to obtain posture information, which is costly and has a cumbersome processing flow.

[0078] Based on this, embodiments of the present application provide a posture estimation method, apparatus, storage medium, and electronic device that help improve posture estimation efficiency. The posture estimation method includes: obtaining a query image, which is used to query the posture information of a target object; when the target object is an unknown target, inputting the query image into a large-scale model, and extracting a target image corresponding to the target object in the query image based on the large-scale model; determining a reference image similar to the target image from a reference image set; and determining the posture information of the target object based on the posture information of the reference image.

[0079] To better understand the technical solutions provided in the embodiments of this application, the following briefly introduces the application scenarios to which the technical solutions provided in the embodiments of this application are applicable. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of this application and are not limiting. In specific implementation, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs.

[0080] FIG1 illustrates an application scenario of the posture estimation method provided by an embodiment of the present application. For example, the application scenario can be an unmanned grasping scenario such as a smart construction site. As shown in FIG1 , the application scenario includes a robot 10, an image processing device 20, and a target object 30.

[0081] The image processing device 20 may be a terminal device or a server, and data may be transmitted between the robot 10 and the image processing device 20 via a communication network.

[0082] The robot 10 collects images of the surrounding environment to obtain a query image, wherein the query image is used to query the posture information of the target object 30. The robot 10 sends the collected query image to the image processing device 20.

[0083] Image processing device 20 extracts the target image corresponding to target object 30 in the query image and identifies reference images similar to the target image from a pre-set reference image set. Image processing device 20 can determine the pose information of the target object based on the pose information of the reference images. During unmanned object capture, the pose information of the target object can be determined simply by acquiring the query image, simplifying the pose estimation process and improving its efficiency.

[0084] As shown in FIG2 , the robot 10 may include a main unit, a robotic arm, and an image collector. The robotic arm is equipped with an image collector, which may be an RGB camera. The image collector can monitor the 6D posture of a target object in real time as the robotic arm moves toward or away from it.

[0085] Robot 10 can use an image collector to capture a query image in real time and send the captured query image to a control device, which then sends the query image to image processing device 20. Image processing device 20 determines the pose information of the target object based on the received query image and sends the pose information to the robot host. The robot host controls the robotic arm to grasp the target object based on the received pose information.

[0086] FIG3 shows another application scenario of the posture estimation method provided in an embodiment of the present application. For example, the application scenario can be an intelligent lifting scenario. As shown in FIG3 , the application scenario includes a crane 10′, a load 20′, an image collector 30′, and an image processing device 40′. The crane 10′ can be a truck crane, a crawler crane, a tower crane, or a rail-mounted crane, and the crane 10′ includes a boom and a hook.

[0087] The image acquisition device 30' can be an RGB camera; the image processing device 40' can be a terminal device or a server. Data can be transmitted between the crane 10' and the image processing device 40' via a communication network. Data can be transmitted between the image acquisition device 30' and the image processing device 40' via a communication network.

[0088] The image collector 30' collects the surrounding environment and obtains a query image containing the hook. The query image is used to query the posture information of the hook. The image collector 30' sends the query image to the image processing device 40'.

[0089] The image processing device 40′ extracts the target image corresponding to the hook in the query image and identifies reference images similar to the target image from a pre-set reference image set. The image processing device 40′ can determine the hook's posture information based on the posture information of the reference images. The image processing device 40′ transmits the determined hook posture information to the crane 10′.

[0090] The crane 10 ′ controls the crane 10 ′ to approach and lift the load with a correct hook posture based on the received hook posture information.

[0091] To further illustrate the technical solutions provided by the embodiments of the present application, this is described in detail below with reference to the accompanying drawings and specific implementation methods. Although the embodiments of the present application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. The method may be executed in the order of the methods shown in the embodiments or drawings or in parallel during the actual processing process or when the device is executed.

[0092] The technical solution provided in the embodiment of the present application is described below in conjunction with the application scenarios shown in Figures 1 and 2 and with reference to the flowchart of a posture estimation method shown in Figure 4.

[0093] S401: Acquire a query image, wherein the query image is used to query the posture information of the target object.

[0094] In some scenarios, embodiments of the present application may obtain a query image captured by an image collector. The image collector may be a conventional RGB camera. Exemplarily, the RGB camera captures the surrounding environment to obtain the query image.

[0095] In other scenarios, embodiments of the present application may obtain a query image from a control device, wherein the query image may be acquired by an image collector in the control device, or may be acquired by the control device from the image collector.

[0096] S402: If the target object is an unknown target, the query image is input into the large-scale model, and a target image corresponding to the target object in the query image is extracted based on the large-scale model.

[0097] After obtaining a query image, the embodiment of the present application determines whether the target object to be queried is a known target. If the target object is determined to be an unknown target, the query image can be input into a large-scale model, and target detection can be performed on the query image based on the large-scale model to determine the coordinate frame of the target object in the query image. Then, based on the determined coordinate frame, a target image corresponding to the target object can be extracted from the query image.

[0098] S403: Determine a reference image similar to the target image from the reference image set.

[0099] In some embodiments, before determining a reference image similar to a target image, the embodiments of the present application obtain a reference image set from offline posture information.

[0100] S404: Determine the posture information of the target object according to the posture information of the reference image.

[0101] The embodiment of the present application can determine a reference image similar to the target image in a set of reference images with known posture information, and then determine the posture information of the target object based on the posture information of the determined reference image, thereby simplifying the posture information estimation process and improving the posture information estimation efficiency.

[0102] In the application scenario shown in Figure 1, the robot needs to determine the presence of a target object within the field of view of the image collector during movement. The embodiment of the present application can determine the posture information of the target object based on the query image obtained from the robot and send the determined posture information to the robot.

[0103] In some embodiments, the target object appearing within the image collector's field of view can be either a known target or an unknown target. The target object type can be manually set. For example, when setting the target object for the robot to grasp, the user can set whether the target object is a known target or an unknown target. A known target refers to a target object that appears in the target detector training task, while an unknown target object that does not appear is considered an unknown target. The following describes the process of extracting the target image based on the two aforementioned scenarios.

[0104] Case 1: The target object is a known target.

[0105] In some embodiments, after the embodiment of the present application determines that the target object for posture estimation is a known target, the acquired query image is input into the target detector, target detection is performed on the query image based on the target detector, and the coordinate frame of the target object in the query image is determined.

[0106] If the embodiment of the present application determines that the coordinate frame of the target object is not empty, an image area corresponding to the coordinate frame in the query image is extracted, and the image area is used as the target image.

[0107] If the embodiment of the present application determines that the coordinate frame of the target object is empty, the posture information of the target object is determined to be empty. In some examples, when the embodiment of the present application determines that the posture information of the target object is empty, the robot can be informed that the query image does not contain the target object and the posture information of the target object is empty.

[0108] In order for the target detector to accurately determine the coordinate frame of the target object, the target detector used in the above process needs to be trained in advance. The sample data used by the target detector is the image data corresponding to the known target.

[0109] Specifically, the embodiment of the present application can train the target detector in the following manner:

[0110] During implementation, the present embodiment captures an image containing a target object and uses the image as a sample image in a training dataset. The present embodiment marks the coordinate frame of the target object on the acquired sample image to obtain a sample dataset. The sample dataset includes multiple sample images and the coordinate frame of the target object contained in each sample image.

[0111] In this embodiment, sample images from a sample dataset are fed into an object detector. Based on the YOLOX algorithm, the detector outputs a coordinate frame of the target object. Based on the coordinate frame of the target object and the sample coordinate frame corresponding to the target object, the network parameters of the object detector are adjusted until a trained object detector is obtained. The trained object detector performs well in scenes such as strong and weak light, complex outdoor backgrounds, and long distances.

[0112] The following takes the target object as an example to illustrate the training process of the target detector: The embodiment of the present application can collect 667 image data and mark the coordinate frame of the hook on the acquired image data. The marked coordinate frames can be divided into training coordinate frames and test coordinate frames. Among them, the number of training coordinate frames can be 950, and the number of test coordinate frames can be 92. The embodiment of the present application inputs the image data into the target detector and uses the YOLOX algorithm to train the target detector. When the YOLOX algorithm is used to achieve the mAP on the test coordinate frame reaching the set value, the trained target detector is obtained. For example, when the mAP is as high as 87.7%, it is determined that the training of the first target detector is completed.

[0113] Case 2: The target object is unknown.

[0114] In some embodiments, when the target object is determined to be an unknown object, the embodiment of the present application obtains text information corresponding to the target object. Exemplarily, when the target object is an unknown object, the embodiment of the present application obtains the query image and simultaneously obtains the text information corresponding to the unknown object.

[0115] For example, when the target object that the user sets the robot to grasp is an unknown object, the robot can generate text information corresponding to the target object. If the unknown object is a teacup, the text information corresponding to the teacup can be "This is a picture of a teacup."

[0116] In an embodiment of the present application, text information and a query image can be input into a large-scale model, target detection can be performed on the query image based on the large-scale model, and a coordinate frame of the target object in the query image can be determined.

[0117] In some embodiments, the large-scale model may include a large-scale image-text model and a large-scale object segmentation model. The large-scale image-text model can accept image and text pairs as input and return feature regions of the query target object. The large-scale object segmentation model can accept prompts such as points, coordinate boxes, and masks to segment the target object.

[0118] In an embodiment of the present application, text information and a query image are input into a large-scale image model in a large-scale model, and features of the query image are extracted based on the large-scale image-text model to obtain feature areas of the target object.

[0119] In some embodiments, if the threshold of the feature region output by the large-scale image-text model is higher than a set threshold, the feature region is input into the large-scale object segmentation model. If the threshold of the feature region output by the large-scale image-text model is lower than the set threshold, the coordinate box of the target object is determined to be empty, and the pose information of the target object is determined to be empty.

[0120] In an embodiment of the present application, the obtained feature area of ​​the target object is input into a large-scale all-things segmentation model in a large-scale model, and the feature area is cut based on the large-scale all-things segmentation model to obtain a coordinate frame of the target object.

[0121] In some embodiments, the embodiment of the present application can select the largest feature area from the feature areas output by the large-scale graphic model and input it into the large-scale all-things segmentation model. In addition, the embodiment of the present application can also input the point with the highest confidence built into the largest feature area, the minimum circumscribed rectangular box of the largest feature area, and the mask of the largest feature area as prompt words into the large-scale all-things segmentation model to obtain an accurate target segmentation mask, and take the minimum circumscribed rectangular box as the coordinate box of the target object. Among them, when there are holes in the mask of the feature area, the holes are filled.

[0122] Exemplarily, the embodiments of the present application can utilize the feature extraction capability of a large-scale graphic model to determine the approximate location of the target object, and then accurately segment the approximate location of the target object through a large-scale object segmentation model to obtain the precise coordinate frame of the target object.

[0123] After obtaining the coordinate frame of the target object, the embodiment of the present application extracts the image area corresponding to the coordinate frame in the query image, and uses the image area as the target image.

[0124] In some embodiments, when extracting the image area corresponding to the coordinate frame, to avoid truncation of the target object, the embodiment of the present application may also enlarge the coordinate frame, perform a center crop on the target object, and scale the target object according to the size of (128, 128) to obtain the target image. For example, the enlargement ratio of the coordinate frame may be a 1.1:1 ratio of length to width.

[0125] After obtaining the target image, embodiments of the present application can also sample offline pose information based on the farthest point sampling technique to obtain multiple pose information. The images corresponding to the multiple pose information constitute a reference image set. The offline pose information is pose information of the target object obtained by offline 3D reconstruction in embodiments of the present application.

[0126] In some embodiments, the embodiments of the present application may generate offline posture information through the following steps:

[0127] The embodiment of the present application collects a video sequence containing target objects, wherein the target objects include known targets and unknown targets.

[0128] In some embodiments, the video sequence captured in the embodiments of the present application is a single-background experimental field, and the duration of the captured video sequence is controlled to be approximately one minute. Furthermore, the video sequence captured in the embodiments of the present application needs to include various perspectives of the target object, and the target object should be captured from a distance to a close distance under the same perspective before switching perspectives.

[0129] After obtaining the video sequence, the embodiment of the present application can use the SFM technology to perform three-dimensional reconstruction on the video sequence to obtain offline posture information of the target object in the video sequence.

[0130] Taking the colmap algorithm as an example, the process of 3D reconstruction using SFM technology is explained: As shown in Figure 5, the embodiment of the present application writes the width, height, and name of each frame in the video sequence into database.db, and gradually performs feature extraction, feature matching, incremental reconstruction, image dedistortion, depth estimation, and other steps on each frame in the video sequence to obtain the 3D point cloud reconstruction effect and the camera's intrinsic parameters, extrinsic parameters, and offline pose information of the target object in each frame. For example, when the target object is a hook, the 3D point cloud reconstruction effect obtained by the embodiment of the present application is shown in Figure 6.

[0131] Considering the insufficient edge computing resources in construction site scenarios, in the posture estimation process, in order to cover the posture information of the target object from various perspectives as much as possible, the farthest point sampling technology (FPS) can be used to downsample the obtained offline posture information.

[0132] In the embodiment of the present application, the offline posture information of the target object in the video sequence can be downsampled based on the farthest point sampling technology to obtain multiple posture information, and the images corresponding to the multiple posture information constitute a reference image set.

[0133] In some embodiments, the embodiments of the present application may downsample offline posture information using the following formula:

[0134] Among them, the initial sample of offline posture information is set as Sampling point N, {s i} is the offline pose information of the target object in each frame; d(q,S) is the maximum distance from the current selected point to the set S, and d(p,S) is the maximum distance from the next point to the set S. Distance can be Euclidean distance, absolute distance, etc.

[0135] The present embodiment utilizes FPS technology to downsample the pose information of the offline target object in each frame of the obtained video sequence to the pose information of N reference images. Furthermore, FPS technology maximizes the preservation of the original data information, significantly reducing computing resources and the time required for subsequent matching and refinement, and shortening the time required to complete a single inference of pose estimation.

[0136] In some embodiments, the embodiment of the present application can also perform farthest point sampling processing on the offline posture information in the offline stage to obtain a reference image set. The embodiment of the present application can directly obtain the reference image set obtained in the offline stage in the posture estimation stage.

[0137] After obtaining the reference image set, the embodiment of the present application can also rotate each reference image in the plane at angles of -90°, -45°, 0°, 45°, and 90° to obtain more reference images that match the query image.

[0138] After obtaining a reference image set, the embodiment of the present application can determine a reference image similar to the target image from the reference image set.

[0139] In some embodiments, the embodiments of the present application can respectively determine the similarity between each reference image in the reference image set and the target image, and use the reference image corresponding to the maximum value of the similarity as the reference image similar to the target image.

[0140] In some examples, embodiments of the present application may input a target image obtained by a target detector or a large-scale model into a matcher selector. Embodiments of the present application may also obtain a reference image set generated offline and input the reference image set into the selector. Embodiments of the present application may calculate the similarity between the target image and reference images in the reference image set through a neural network in the selector to obtain a reference image that is most similar to the target image.

[0141] In the embodiment of the present application, after obtaining a reference image similar to the target image, the posture information of the target object is determined based on the posture information of the reference image.

[0142] In some embodiments, the present invention can use the pose information of a reference image similar to the target image as the initial pose information of the target image. The present invention can also input the obtained initial pose information into a refiner to converge the initial pose information to obtain the pose information of the target object.

[0143] After obtaining the initial posture information, the embodiment of the present application determines multiple posture information similar to the initial posture information from the posture information corresponding to each reference image in the reference image set.

[0144] The embodiment of the present application can perform regression processing on the query image, the reference images corresponding to the multiple posture information, the multiple posture information and the initial posture information to obtain the posture residual corresponding to the query image, and update the initial posture information according to the posture residual to obtain the posture information of the target object.

[0145] In some embodiments, the present invention can input the reference image corresponding to the obtained multiple pose information, the multiple pose information, the query image, and the initial pose information into a refiner to regress the pose residual in this state. The refiner can be a 3D convolutional neural network (CNN).

[0146] In an embodiment of the present application, a reference image, multiple pose information, a query image, and initial pose information corresponding to multiple pose information can be input into a 3D CNN, and the inconsistencies among the reference image, multiple pose information, the query image, and the initial pose information corresponding to the multiple pose information can be analyzed based on the 3D CNN, and the pose residual can be obtained by regression.

[0147] In an embodiment of the present application, the initial pose information can be updated based on the pose residual to obtain the pose information of the target object. For example, in an embodiment of the present application, an iteration parameter in the refiner can be set, for example, the iteration parameter can be set to 3. In another example, in an embodiment of the present application, the initial pose information can be iteratively updated based on the pose residual until the obtained pose residual is less than a set threshold, and the pose information of the target object is output.

[0148] For example, an embodiment of the present application inputs the initial posture information, the six posture information most similar to the initial posture information in the posture information corresponding to the reference image set, the reference images corresponding to the six posture information, and the query image into the refiner, regresses the posture residual in this state, updates the initial posture information with the posture residual, performs three iterations, and outputs the posture information of the target object.

[0149] In some embodiments, when the query image is a single image, the embodiments of the present application may send the posture information obtained above to the control device so that the control device can capture the target object according to the posture information.

[0150] The control device may be a robot, a robot host in a robot, or a crane.

[0151] In some embodiments, when the query image is a frame image in a video sequence, the present application may also implement a temporal smoothing method to solve the problem of jitter in the envelope cuboid generated by the posture information when the video is continuously generated.

[0152] After obtaining the posture information of the target object, the embodiment of the present application can also obtain the posture information of multiple consecutive frames of images from the video sequence corresponding to the query image, wherein the last frame of the multiple frames is the previous frame of the query image.

[0153] For example, an embodiment of the present application may obtain posture information of five consecutive frames of images preceding a query image in a video sequence.

[0154] The embodiment of the present application may also perform weighted averaging on the posture information of the target object and the posture information of multiple frames of images to obtain updated initial posture information of the target object.

[0155] In some embodiments, the present application embodiment can obtain the updated initial posture information in the following manner: t =aPr t +bPr t-1 +cPr t-2 +dPr t-3 +ePr t-4

[0156] Among them, a, b, c, d, e are the weight coefficients of different previous and next frames, which add up to 1; t represents the number of the current frame; Pr` t represents the updated initial pose information of the query image; Pr t Represents the pose information of the target object in the query image; Pr t-1 Indicates the pose information of the target object in the previous frame of the query image; Pr t-2 Pr represents the pose information of the target object in the previous two frames of the query image; t-3 Pr represents the pose information of the target object in the last three frames of the query image; t-4 Indicates the pose information of the target object in the previous 4 frames of the query image.

[0157] After obtaining the updated initial pose information, the embodiment of the present application determines multiple pose information similar to the updated initial pose information from the pose information corresponding to each reference image in the reference image set. The embodiment of the present application may also perform regression processing on the multiple pose information, the reference images corresponding to the multiple pose information, the query image, and the updated initial pose information to obtain a pose residual of the query image, and update the obtained updated initial pose information based on the obtained pose residual to obtain updated pose information of the target object.

[0158] For example, embodiments of the present application may further input the updated initial pose information, multiple pose information similar to the updated initial pose information, reference images corresponding to the multiple pose information, and a query image into a refiner to obtain updated pose information of the target object. This process is the same as the aforementioned process of inputting the initial pose information, multiple pose information similar to the initial pose information, reference images corresponding to the multiple pose information, and a query image into a refiner to obtain pose information of the target object, and is not further described here.

[0159] In some embodiments, the present application provides a specific use case of a pose estimation method. As shown in Figure 7, after acquiring a query image, the present application inputs the query image into an object detector. The query image is used to query the pose information of a target object, which is a known target.

[0160] In the embodiment of the present application, a target detector is used to perform target detection on the query image to determine whether there is a target object in the query image.

[0161] In some embodiments, when a target object is present in a query image, a target image corresponding to the target object in the query image is input into the matcher. Embodiments of the present application may also improve the matcher's sampling of offline pose information to obtain a reference image set. Embodiments of the present application utilize the matcher to select a reference image from the reference image set that is most similar to the target image, and use the pose information of the reference image as the initial pose information for the target image.

[0162] In embodiments of the present application, the initial pose information, the query image, and the pose information corresponding to the reference image set, along with multiple pose information similar to the initial pose information, and the reference images corresponding to the multiple pose information, can be input into a refiner, and the refiner performs regression processing to obtain the pose information of the target object. Exemplarily, the refiner can be a 3D CNN.

[0163] After obtaining the posture information of the target object, the embodiment of the present application outputs the posture information of the target object. In some embodiments, when there is no target object in the query image, the embodiment of the present application determines that the posture information of the target object is empty and outputs the posture information.

[0164] In some embodiments, the present application provides a specific use case for another pose estimation method. As shown in Figure 8, after acquiring a query image, the present application determines that the target object to be queried is an unknown target and obtains the text information corresponding to the target object. The present application inputs the query image and text information into a large-scale image-text model within a large-scale model, and performs object detection on the query image based on the large-scale image-text model.

[0165] When there is no target object in the query image, the embodiment of the present application determines that the posture information of the target object is empty and outputs the posture information.

[0166] When there is a target object in the query image, the embodiment of the present application outputs the feature area of ​​the target object through the large-scale graphic model, and inputs the feature area into the large-scale all-things segmentation model. The feature area is cut based on the large-scale all-things segmentation model to obtain the coordinate frame of the target object. The embodiment of the present application extracts the target image of the target object based on the coordinate frame, and inputs the target image into the matcher to determine the initial posture information of the target image. This process is the same as the process of determining the initial posture information by the matcher in Figure 7, and will not be repeated here.

[0167] In an embodiment of the present application, the process of inputting the initial posture information, multiple posture information similar to the initial posture information in the posture information corresponding to the reference image set, the reference images corresponding to the multiple posture information, and the query image into the refiner to obtain the posture information of the target object is the same as the process of obtaining the posture information of the target object through the refiner in Figure 7, and will not be repeated here.

[0168] Based on the content described in Figure 8, the embodiment of the present application can input the query image and the text information corresponding to the target object into a large-scale image-text model, detect the unknown target based on the large-scale image-text model, and obtain the target image of the unknown target. The posture information of the unknown target can be obtained without training the unknown image. Therefore, the posture estimation method provided by the embodiment of the present application has strong generalization. In addition, the present application can also determine the posture information of the target object based on the posture information of the reference image obtained offline, thereby reducing the complexity of the posture estimation process and improving the estimation efficiency.

[0169] As shown in FIG9 , an embodiment of the present application provides a complete flow diagram of a posture estimation method, wherein the target object is a known target as an example, including the following steps:

[0170] S901: Acquire a query image, wherein the query image is used to query the posture information of the target object.

[0171] S902: Input the query image into the object detector, perform object detection on the query image based on the object detector, and determine the coordinate box of the target object in the query image.

[0172] In some embodiments, when determining the coordinate frame of the target object, the embodiment of the present application also needs to determine whether the coordinate frame is empty. If the coordinate frame is empty, the posture information of the target object is determined to be empty. If the coordinate frame is not empty, step S903 is executed.

[0173] S903: Extracting an image region corresponding to the coordinate box in the query image, and using the image region as the target image.

[0174] S904: Determine the similarity between each reference image in the reference image set and the target image.

[0175] In some embodiments, before determining the similarity between each reference image in the reference image set and the target image, embodiments of the present application may further sample offline pose information based on a farthest point sampling technique to obtain multiple pose information. The images corresponding to the multiple pose information constitute the reference image set. The offline pose information is pose information of the target object obtained through offline 3D reconstruction.

[0176] S905: The reference image corresponding to the maximum value of the similarity is used as the reference image similar to the target image.

[0177] S906: Taking the posture information of the reference image similar to the target image as the initial posture information of the target image.

[0178] S907: Determine, from the posture information corresponding to each reference image in the reference image set, a plurality of posture information similar to the initial posture information.

[0179] S908: Perform regression processing on the query image, the reference images corresponding to the multiple pose information, the multiple pose information, and the initial pose information to obtain a pose residual corresponding to the query image.

[0180] S909: Update the initial posture information according to the posture residual to obtain the posture information of the target object.

[0181] S910: Determine whether the query image is a single image; if so, execute step S911; if not, execute step S912.

[0182] S911: Sending the posture information of the target object to the control device, so that the control device grasps the target object according to the posture information of the target object.

[0183] S912: Acquire posture information of multiple consecutive frames of images from the video sequence corresponding to the query image.

[0184] The last frame of the multiple frames is the previous frame of the query image.

[0185] S913: Perform weighted averaging on the posture information of the target object and the posture information of multiple frames of images to obtain updated initial posture information of the target object.

[0186] S914: Determine, from the posture information corresponding to each reference image in the reference image set, a plurality of posture information similar to the updated initial posture information.

[0187] S915: Perform regression processing on the query image, the reference images corresponding to the multiple pose information, the multiple pose information, and the updated initial pose information to obtain a pose residual corresponding to the query image.

[0188] S916: Obtain updated initial posture information obtained according to the posture residual update to obtain updated posture information of the target object.

[0189] S917: Send the updated posture information to the control device, so that the control device grabs the target object according to the updated posture information.

[0190] Based on the content described in Figure 9, the embodiment of the present application can determine the pose information of the target object based on the pose information of reference images similar to the target image in the reference image set, without the need for online 3D reconstruction. This simplifies the pose estimation process, improves the efficiency of pose estimation, and ensures the accuracy of pose estimation. In addition, the embodiment of the present application can also perform pose estimation for known and unknown targets, improving generalization.

[0191] Based on the same inventive concept, an embodiment of the present application also provides a posture estimation device. Since the principle of solving the problem by the device is similar to that of the above-mentioned posture estimation method, the device can be implemented with reference to the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0192] As shown in FIG10 , an embodiment of the present application provides a structural block diagram of a posture estimation device, which includes:

[0193] The communication unit 1001 is used to obtain a query image, where the query image is used to query the posture information of the target object;

[0194] an extraction unit 1002 configured to, if the target object is an unknown target, input the query image into a large-scale model, and extract a target image corresponding to the target object in the query image based on the large-scale model;

[0195] The determining unit 1003 is configured to determine a reference image similar to the target image from a reference image set; and determine the posture information of the target object according to the posture information of the reference image.

[0196] In an optional implementation manner, the extraction unit 1002 is specifically configured to:

[0197] Obtaining text information corresponding to the target object;

[0198] Inputting the text information and the query image into the large-scale model, performing target detection on the query image based on the large-scale model, and determining a coordinate frame of the target object in the query image;

[0199] An image region corresponding to the coordinate frame in the query image is extracted, and the image region is used as the target image.

[0200] In an optional implementation, the large-scale model includes a large-scale image and text model and a large-scale object segmentation model; the extraction unit 1002 is specifically configured to:

[0201] Inputting the text information and the query image into the large-scale image-text model, performing feature extraction on the query image based on the large-scale image-text model, and determining a feature region of the target object;

[0202] The feature region is input into the large-scale all-things segmentation model, and the feature region is cut based on the large-scale all-things segmentation model to obtain a coordinate frame of the target object.

[0203] In an optional implementation manner, the extraction unit 1002 is further configured to:

[0204] If the target object is a known target, inputting the query image into a target detector, performing target detection on the query image based on the target detector, and determining a coordinate frame of the target object in the query image;

[0205] An image region corresponding to the coordinate frame in the query image is extracted, and the image region is used as the target image.

[0206] In an optional implementation manner, the determining unit 1003 is specifically configured to:

[0207] Based on the farthest point sampling technology, the offline posture information is sampled to obtain multiple posture information, and the images corresponding to the multiple posture information constitute the reference image set; the offline posture information is the posture information of the target object obtained by three-dimensional reconstruction in an offline state;

[0208] respectively determining a similarity between each reference image in the reference image set and the target image;

[0209] The reference image corresponding to the maximum value of the similarities is used as a reference image similar to the target image.

[0210] In an optional implementation manner, the determining unit 1003 is specifically configured to:

[0211] Using the pose information of a reference image similar to the target image as the initial pose information of the target image;

[0212] Determining, from the posture information corresponding to each reference image in the reference image set, a plurality of posture information similar to the initial posture information;

[0213] Performing regression processing on the query image, the reference images corresponding to the multiple pose information, the multiple pose information, and the initial pose information to obtain a pose residual corresponding to the query image;

[0214] The initial posture information is updated according to the posture residual to obtain the posture information of the target object.

[0215] In an optional implementation manner, after obtaining the posture information of the target object, the determining unit 1003 is further configured to:

[0216] If the query image is a frame image in a video sequence, then obtaining posture information of multiple consecutive frames of images from the video sequence, wherein the last frame image in the multiple frames of images is the previous frame image of the query image;

[0217] Performing a weighted average on the posture information of the target object and the posture information of the multiple frames of images to obtain updated initial posture information of the target object;

[0218] Determining, from the posture information corresponding to each reference image in the reference image set, a plurality of posture information similar to the updated initial posture information;

[0219] Performing regression processing on the query image, the reference images corresponding to the multiple pose information, the multiple pose information, and the updated initial pose information to obtain a pose residual corresponding to the query image;

[0220] The updated initial posture information is updated according to the posture residual to obtain updated posture information of the target object.

[0221] In an optional implementation manner, the communication unit 1001 is further configured to:

[0222] The posture information of the target object is sent to a control device, so that the control device grasps the target object according to the posture information.

[0223] Corresponding to the embodiment of the posture estimation method described above, an embodiment of the present application further provides an electronic device. The electronic device may be a server, or a terminal such as a mobile terminal or a computer. Exemplarily, the electronic device may be an industrial control device.

[0224] The electronic device includes at least a memory for storing data and a processor for data processing. Among them, for the processor for data processing, when performing processing, a microprocessor, a CPU, a GPU (Graphics Processing Unit), a DSP or an FPGA can be used. For the memory, the memory stores operating instructions, which can be computer executable codes, and the various steps in the process of the posture estimation method of the embodiment of the present application are implemented by the operating instructions.

[0225] Figure 11 is a structural schematic diagram of an electronic device provided in an embodiment of the present application; as shown in Figure 11, the electronic device 100 in the embodiment of the present application includes: a processor 101, a display 102, a memory 103, an input device 106, a bus 105 and a communication module 104; the processor 101, the memory 103, the input device 106, the display 102 and the communication module 104 are all connected through the bus 105, and the bus 105 is used to transmit data between the processor 101, the memory 103, the display 102, the communication module 104 and the input device 106.

[0226] Among them, the memory 103 can be used to store software programs and modules, such as the program instructions / modules corresponding to the posture estimation method in the embodiment of the present application. The processor 101 executes various functional applications and data processing of the electronic device 100 by running the software programs and modules stored in the memory 103, such as the posture estimation method provided in the embodiment of the present application. The memory 103 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application application, etc.; the data storage area may store data created according to the use of the electronic device 100 (such as reference image sets and other related data), etc. In addition, the memory 103 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0227] The processor 101 is the control center of the electronic device 100. It connects various parts of the entire electronic device 100 using the bus 105 and various interfaces and lines. It performs various functions of the electronic device 100 and processes data by running or executing software programs and / or modules stored in the memory 103 and calling data stored in the memory 103. Optionally, the processor 101 may include one or more processing units, such as a CPU, a GPU, a digital processing unit, etc.

[0228] The processor 101 may display the result of the posture estimation to the user via the display 102 .

[0229] The processor 101 may also connect to a network via the communication module 104 to obtain a query image and the like.

[0230] The input device 106 is primarily used to obtain user input operations. Depending on the electronic device, the input device 106 may also vary. For example, if the electronic device is a computer, the input device 106 may be a mouse, keyboard, or other input device; if the electronic device is a portable device such as a smartphone or tablet, the input device 106 may be a touch screen.

[0231] Embodiments of the present application also provide a computing device-readable storage medium for posture estimation methods, wherein the contents are not lost after a power outage. The storage medium stores a software program, including program code. When the program code is executed on a computing device, the software program, when read and executed by one or more processors, implements any of the posture estimation methods described in the embodiments of the present application.

[0232] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0233] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.

[0234] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0235] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0236] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A posture estimation method, the method comprising: Acquire a query image, where the query image is used to query posture information of a target object; If the target object is an unknown target, inputting the query image into a large-scale model, and extracting a target image corresponding to the target object in the query image based on the large-scale model; Determining a reference image similar to the target image from a reference image set; The posture information of the target object is determined according to the posture information of the reference image.

2. The method according to claim 1, characterized in that Inputting the query image into a large-scale model, and extracting a target image corresponding to the target object in the query image based on the large-scale model, comprises: Obtaining text information corresponding to the target object; Inputting the text information and the query image into the large-scale model, performing target detection on the query image based on the large-scale model, and determining a coordinate frame of the target object in the query image; An image region corresponding to the coordinate frame in the query image is extracted, and the image region is used as the target image.

3. The method according to claim 2, wherein: The large-scale model includes a large-scale image-text model and a large-scale object segmentation model; the text information and the query image are input into the large-scale model, target detection is performed on the query image based on the large-scale model, and a coordinate frame of the target object in the query image is determined, including: Inputting the text information and the query image into the large-scale image-text model, performing feature extraction on the query image based on the large-scale image-text model, and determining a feature region of the target object; The feature region is input into the large-scale all-things segmentation model, and the feature region is cut based on the large-scale all-things segmentation model to obtain a coordinate frame of the target object.

4. The method according to claim 1, further comprising: If the target object is a known target, inputting the query image into a target detector, performing target detection on the query image based on the target detector, and determining a coordinate frame of the target object in the query image; An image region corresponding to the coordinate frame in the query image is extracted, and the image region is used as the target image.

5. The method according to any one of claims 1 to 4, wherein: The determining of a reference image similar to the target image from a reference image set comprises: Based on the farthest point sampling technology, the offline posture information is sampled to obtain multiple posture information, and the images corresponding to the multiple posture information constitute the reference image set; the offline posture information is the posture information of the target object obtained by three-dimensional reconstruction in an offline state; Determining respectively the similarity between each reference image in the reference image set and the target image; The reference image corresponding to the maximum value of the similarities is used as a reference image similar to the target image.

6. The method according to claim 5, wherein: The step of determining the posture information of the target object according to the posture information of the reference image includes: Using the pose information of a reference image similar to the target image as the initial pose information of the target image; Determining, from the posture information corresponding to each reference image in the reference image set, a plurality of posture information similar to the initial posture information; Performing regression processing on the query image, the reference images corresponding to the plurality of posture information, the plurality of posture information and the initial posture information to obtain a posture residual corresponding to the query image; The initial posture information is updated according to the posture residual to obtain the posture information of the target object.

7. The method according to claim 6, after obtaining the posture information of the target object, the method further comprises: If the query image is a frame image in a video sequence, the posture information of multiple consecutive frames of images is obtained from the video sequence, and the last frame image in the multiple frames is the query image. The previous frame image; Performing weighted averaging on the posture information of the target object and the posture information of the multiple frames of images to obtain updated initial posture information of the target object; Determining, from the posture information corresponding to each reference image in the reference image set, a plurality of posture information similar to the updated initial posture information; Performing regression processing on the query image, the reference images corresponding to the plurality of posture information, the plurality of posture information and the updated initial posture information to obtain a posture residual corresponding to the query image; The updated initial posture information is updated according to the posture residual to obtain updated posture information of the target object.

8. The method according to claim 6, further comprising: The posture information of the target object is sent to a control device, so that the control device grabs the target object according to the posture information.

9. A posture estimation device, comprising: A communication unit, used to obtain a query image, wherein the query image is used to query the posture information of the target object; an extraction unit, configured to input the query image into a large-scale model if the target object is an unknown target, and extract a target image corresponding to the target object in the query image based on the large-scale model; The determination unit is used to determine a reference image similar to the target image from a reference image set; and determine the posture information of the target object according to the posture information of the reference image.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images