A posture restoration method and device

Abstract

This application provides a pose restoration method and apparatus, relating to the field of visual positioning technology, comprising: acquiring a target image of a target scene captured by a camera in an electronic device and target depth data captured by a sensor; determining the initial pose of the electronic device and the initial pose of a second feature point within the camera's field of view in the target scene based on the target image and a previous frame image of the target image; extracting a reference image containing a third feature point from the target image based on the image area occupied by the first feature point in the target image; calculating scale restoration parameters of the target image based on the target depth data of the third feature point in the reference image and the initial pose of the third feature point; and restoring the initial pose of the electronic device and the initial pose of the second feature point according to the scale restoration parameters to obtain the target pose of the electronic device and the target pose of the second feature point, thereby improving the accuracy of the determined target pose.

Description

Technical Field

[0001] This application relates to the field of visual positioning technology, and in particular to a pose restoration method and apparatus. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) technology refers to the process by which a mobile robot, while moving in an unknown environment, collects data about the scene using sensors installed on the robot. Based on this data, it determines the robot's own pose and the poses of various feature points in the scene, and then creates a map of the scene according to the poses of these feature points. SLAM technology is widely used in various technical fields, enabling robots to achieve autonomous localization and navigation.

[0003] However, due to environmental factors and other influences, the accuracy of the determined pose of the mobile robot and the pose of each feature point in the scene is relatively low. Summary of the Invention

[0004] The purpose of this application is to provide a pose repair method and apparatus to improve the accuracy of the determined target pose of an electronic device and the target pose of feature points in a target scene. The specific technical solution is as follows:

[0005] Firstly, in order to achieve the above objectives, embodiments of this application provide a pose restoration method, which is applied to an electronic device, the electronic device including a camera and a sensor, and the method includes:

[0006] The system acquires a target image of the target scene captured by the camera and target depth data of the target scene captured by the sensor; wherein the sensor is located within the field of view of the camera; the target depth data includes the distance between a first feature point within the sensor's acquisition range in the target scene and the electronic device.

[0007] Based on the target image and the previous frame image of the target image, the initial pose of the current electronic device and the initial pose of the second feature point within the camera's field of view in the target scene are determined; wherein, the initial pose of the electronic device is the three-dimensional coordinate of the electronic device in the world coordinate system; the initial pose of the second feature point is the three-dimensional coordinate of the second feature point in the world coordinate system;

[0008] Based on the image region occupied by the first feature point in the target image, a reference image containing a third feature point is extracted from the target image; wherein, the third feature point belongs to the first feature point;

[0009] Based on the target depth data of the third feature point in the reference image and the current initial pose of the third feature point, the scale restoration parameters of the target image are calculated.

[0010] According to the scale repair parameters, the initial pose of the current electronic device and the initial pose of the second feature point are repaired to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0011] Optionally, calculating the scale restoration parameters of the target image based on the target depth data of the third feature point in the reference image and the current initial pose of the third feature point includes:

[0012] Calculate the average distance between the third feature point and the electronic device to obtain the first average distance;

[0013] Calculate the mean of the coordinate values ​​of the third feature point on a specified coordinate axis in the world coordinate system, and use it as the second distance mean;

[0014] The ratio of the first mean distance to the second mean distance is calculated to obtain the scale restoration parameters of the target image.

[0015] Optionally, the step of repairing the initial pose of the current electronic device and the initial pose of the second feature point according to the scale repair parameters to obtain the target pose of the current electronic device and the target pose of the second feature point includes:

[0016] The product of the initial pose of the current electronic device and the scale repair parameter, and the product of the initial pose of the second feature point and the scale repair parameter are calculated respectively to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0017] Optionally, extracting a reference image containing a third feature point from the target image based on the image region occupied by the first feature point in the target image includes:

[0018] According to the pre-recorded location of the calibration area, the image area occupied by each first feature point in the target image is determined to obtain candidate images; wherein, the location of the calibration area is determined by calibration based on the image to be processed pre-acquired by the camera;

[0019] Target detection is performed on the target image to obtain the image region occupied by each target object in the target image, thus obtaining the object image;

[0020] The candidate image is compared with the object image. If the candidate image contains a target object, the image to be processed is determined as the reference image.

[0021] If the candidate image includes multiple target objects, the image region occupied by the target object with the largest area is extracted from the candidate image to obtain a reference image.

[0022] Optionally, before extracting a reference image containing the third feature point from the target image based on the image region occupied by the first feature point in the target image, the method further includes:

[0023] The system acquires an image of the calibration board placed in the target scene, captured by the camera, and depth data of the calibration board, captured by the sensor; wherein the depth data of the calibration board includes the distance between each feature point in the calibration board and the electronic device.

[0024] The image region occupied by the calibration plate is extracted from the image to be processed to obtain the calibration image;

[0025] Based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image, and the camera parameters of the camera, the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system are determined.

[0026] Based on the distance between each feature point in the calibration board and the electronic device, and the three-dimensional coordinates of each feature point in the world coordinate system, the calibration parameters of the calibration image are calculated.

[0027] If the calibration parameter is not less than the first threshold, a new calibration image is determined from the image to be processed, and the process returns to the step of determining the three-dimensional coordinates of each feature point in the calibration board in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration board in the calibration image and the camera parameters of the camera.

[0028] If the calibration parameter is less than the first threshold, the position of the current calibration image in the image to be processed is determined, and the position of the calibration region is obtained.

[0029] Optionally, determining a new calibration image from the image to be processed includes:

[0030] In the image to be processed, the previously determined calibration image is offset according to a preset offset to obtain a new calibration image.

[0031] Optionally, calculating the calibration parameters of the calibration image based on the distance between each feature point on the calibration board and the electronic device, and the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system, includes:

[0032] Based on the distance between each feature point on the calibration board and the electronic device, and the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system, and a first preset formula, the calibration parameters of the calibration image are calculated; wherein, the first preset formula is:

[0033]

[0034] Δd represents the calibration parameters of the calibration image; The z represents the average distance between each feature point in the calibration board and the electronic device; N represents the number of feature points in the calibration board; z i This represents the coordinate value of the i-th feature point in the calibration plate on a specified coordinate axis in the world coordinate system.

[0035] Optionally, determining the initial pose of the current electronic device and the initial pose of the second feature point within the camera's field of view in the target scene based on the target image and the previous frame image of the target image includes:

[0036] For each second feature point in the target image, a feature descriptor for the second feature point is generated based on the distance between the second feature point and other feature points within its neighborhood.

[0037] For each fourth feature point in the previous frame of the target image, a feature descriptor for the fourth feature point is generated based on the distance between the fourth feature point and other feature points within the neighborhood of the fourth feature point.

[0038] Based on the feature descriptors of each second feature point and each fourth feature point, calculate the matching relationship between each second feature point and each fourth feature point;

[0039] Based on the matching relationship between each second feature point and each fourth feature point, the transformation relationship from the previous frame image of the target image to the target image is calculated, and the initial pose of the current electronic device is determined based on the transformation relationship.

[0040] Based on the two-dimensional coordinates of each second feature point in the target image and the camera parameters of the camera, the three-dimensional coordinates of each second feature point in the world coordinate system are calculated to obtain the initial pose of the second feature points within the field of view of the camera in the target scene.

[0041] Optionally, the electronic device is a mobile robot; the camera is a monocular camera; the optical axis of the camera is parallel to the horizontal plane; the sensor is a single-point TOF sensor; and the sensor is located directly above or below the camera.

[0042] Secondly, in order to achieve the above objectives, embodiments of this application provide a pose restoration device, which is applied to an electronic device, the electronic device including a camera and a sensor, and the device includes:

[0043] The first acquisition module is used to acquire the target image of the target scene currently captured by the camera, and the target depth data of the target scene currently captured by the sensor; wherein the sensor is located within the field of view of the camera; the target depth data includes the distance between a first feature point within the sensor's acquisition range in the target scene and the electronic device;

[0044] An initial pose determination module is used to determine the initial pose of the current electronic device and the initial pose of a second feature point within the camera's field of view in the target scene based on the target image and the previous frame image of the target image; wherein, the initial pose of the electronic device is the three-dimensional coordinates of the electronic device in the world coordinate system; and the initial pose of the second feature point is the three-dimensional coordinates of the second feature point in the world coordinate system.

[0045] A reference image acquisition module is used to extract a reference image containing a third feature point from the target image based on the image area occupied by the first feature point in the target image; wherein the third feature point belongs to the first feature point;

[0046] The scale restoration parameter determination module is used to calculate the scale restoration parameters of the target image based on the target depth data of the third feature point in the reference image and the current initial pose of the third feature point.

[0047] The target pose determination module is used to repair the initial pose of the current electronic device and the initial pose of the second feature point according to the scale repair parameters, so as to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0048] Optionally, the scale repair parameter determination module is specifically used to calculate the mean distance between the third feature point and the electronic device to obtain a first mean distance.

[0049] Calculate the mean of the coordinate values ​​of the third feature point on a specified coordinate axis in the world coordinate system, and use it as the second distance mean;

[0050] The ratio of the first mean distance to the second mean distance is calculated to obtain the scale restoration parameters of the target image.

[0051] Optionally, the target pose determination module is specifically used to calculate the product of the initial pose of the current electronic device and the scale repair parameter, and the product of the initial pose of the second feature point and the scale repair parameter, respectively, to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0052] Optionally, the reference image acquisition module is specifically used to determine the image region occupied by each first feature point in the target image according to the position of the pre-recorded calibration region, so as to obtain a candidate image; wherein, the position of the calibration region is determined by calibration based on the image to be processed pre-acquired by the camera;

[0053] Target detection is performed on the target image to obtain the image region occupied by each target object in the target image, thus obtaining the object image;

[0054] The candidate image is compared with the object image. If the candidate image contains a target object, the image to be processed is determined as the reference image.

[0055] If the candidate image includes multiple target objects, the image region occupied by the target object with the largest area is extracted from the candidate image to obtain a reference image.

[0056] Optionally, the device further includes:

[0057] The second acquisition module is used to acquire, before the reference image acquisition module performs the extraction of a reference image containing a third feature point from the target image based on the image region occupied by the first feature point in the target image, an image to be processed including a calibration board placed in the target scene captured by the camera, and depth data of the calibration board captured by the sensor; wherein, the depth data of the calibration board includes the distance between each feature point in the calibration board and the electronic device.

[0058] The calibration image acquisition module is used to extract the image region occupied by the calibration plate from the image to be processed, and obtain the calibration image;

[0059] The three-dimensional coordinate determination module is used to determine the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image and the camera parameters of the camera.

[0060] The calibration parameter determination module is used to calculate the calibration parameters of the calibration image based on the distance between each feature point in the calibration board and the electronic device, and the three-dimensional coordinates of each feature point in the world coordinate system.

[0061] The calibration image update module is used to determine a new calibration image from the image to be processed if the calibration parameter is not less than a first threshold, and to trigger the three-dimensional coordinate determination module to perform the step of determining the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image and the camera parameters of the camera.

[0062] The calibration region determination module is used to determine the position of the current calibration image in the image to be processed if the calibration parameter is less than the first threshold, thereby obtaining the position of the calibration region.

[0063] Optionally, the calibration image update module is specifically used to offset the previously determined calibration image in the image to be processed according to a preset offset to obtain a new calibration image.

[0064] Optionally, the calibration parameter determination module is specifically used to calculate the calibration parameters of the calibration image based on the distance between each feature point on the calibration board and the electronic device, the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system, and a first preset formula; wherein, the first preset formula is:

[0065]

[0066] Δd represents the calibration parameters of the calibration image; The z represents the average distance between each feature point in the calibration board and the electronic device; N represents the number of feature points in the calibration board; z i This represents the coordinate value of the i-th feature point in the calibration plate on a specified coordinate axis in the world coordinate system.

[0067] Optionally, the initial pose determination module is specifically used to generate a feature descriptor for each second feature point in the target image based on the distance between the second feature point and other feature points within the neighborhood of the second feature point.

[0068] For each fourth feature point in the previous frame of the target image, a feature descriptor for the fourth feature point is generated based on the distance between the fourth feature point and other feature points within the neighborhood of the fourth feature point.

[0069] Based on the feature descriptors of each second feature point and each fourth feature point, calculate the matching relationship between each second feature point and each fourth feature point;

[0070] Based on the matching relationship between each second feature point and each fourth feature point, the transformation relationship from the previous frame image of the target image to the target image is calculated, and the initial pose of the current electronic device is determined based on the transformation relationship.

[0071] Based on the two-dimensional coordinates of each second feature point in the target image and the camera parameters of the camera, the three-dimensional coordinates of each second feature point in the world coordinate system are calculated to obtain the initial pose of the second feature points within the field of view of the camera in the target scene.

[0072] Optionally, the electronic device is a mobile robot; the camera is a monocular camera; the optical axis of the camera is parallel to the horizontal plane; the sensor is a single-point TOF sensor; and the sensor is located directly above or below the camera.

[0073] This application also provides an electronic device, including:

[0074] Memory, used to store computer programs;

[0075] The processor, when executing a program stored in memory, implements any of the pose repair methods described above.

[0076] This application also provides a mobile robot system, including a data acquisition module and a processor; the data acquisition module includes a camera and sensors;

[0077] The data acquisition module is used to acquire target images of the target scene through the camera, and to acquire target depth data of the target scene through the sensor;

[0078] The processor is used to execute any of the pose repair method steps described above.

[0079] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the pose repair methods described above.

[0080] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the pose repair methods described above.

[0081] Beneficial effects of the embodiments in this application:

[0082] This application provides a pose restoration method that acquires a target image of a target scene captured by a camera in a current electronic device, and target depth data of the target scene captured by a current sensor; the sensor is located within the field of view of the camera; the target depth data includes the distance between a first feature point within the sensor's acquisition range in the target scene and the electronic device; based on the target image and the previous frame of the target image, the initial pose of the current electronic device and the initial pose of a second feature point within the camera's field of view in the target scene are determined. The initial pose of the electronic device is the three-dimensional coordinate of the electronic device in the world coordinate system; the initial pose of the second feature point is the three-dimensional coordinate of the second feature point in the world coordinate system; based on the image area occupied by the first feature point in the target image, a reference image containing a third feature point is extracted from the target image; the third feature point belongs to the first feature point; based on the target depth data of the third feature point in the reference image and the initial pose of the current third feature point, scale restoration parameters of the target image are calculated; according to the scale restoration parameters, the initial pose of the current electronic device and the initial pose of the second feature point are restored to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0083] Based on the above processing, the target depth data of the third feature point can represent the pose of the second feature point obtained from the sensor, and the initial pose of the second feature point represents the pose of the third feature point obtained from the camera. Correspondingly, the scale repair parameter calculated based on the target depth data of the third feature point and its initial pose can represent the difference between the pose of the third feature point obtained from the sensor and the pose obtained from the camera. Furthermore, the scale of the pose change of the electronic device, the scale of the pose change of the second feature point, and the scale of the pose change of the third feature point are the same. Therefore, repairing the initial pose of the electronic device and the initial pose of the second feature point in the target scene based on the scale repair parameter can improve the accuracy of the determined target pose of the electronic device and the target pose of the second feature point.

[0084] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0086] Figure 1 A flowchart illustrating the first pose repair method provided in this application embodiment;

[0087] Figure 2 A diagram illustrating the mounting structure of a camera and sensor provided in an embodiment of this application;

[0088] Figure 3 A flowchart illustrating the second pose repair method provided in this application embodiment;

[0089] Figure 4 A flowchart illustrating the third pose repair method provided in this application embodiment;

[0090] Figure 5 A flowchart illustrating the fourth pose repair method provided in this application embodiment;

[0091] Figure 6 A flowchart illustrating the fifth pose repair method provided in this application embodiment;

[0092] Figure 7 A flowchart of the sixth pose repair method provided in the embodiments of this application;

[0093] Figure 8 A structural diagram of a pose restoration device provided in an embodiment of this application;

[0094] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0095] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0096] In related technologies, the accuracy of the pose of the mobile robot and the poses of various feature points in the scene determined during the SLAM process is relatively low.

[0097] To address the aforementioned problems, this application provides a pose restoration method applied to an electronic device, which includes a camera and sensors. The electronic device is a mobile robot, such as a robotic vacuum cleaner or an AGV (Automated Guided Vehicle). The camera in the electronic device can acquire target images of the target scene, and the sensors can acquire target depth data of the target scene. Based on the target images and target depth data of the target scene, scale restoration parameters are calculated. According to the calculated scale restoration parameters, the initial pose of the electronic device and the initial poses of each feature point in the target scene are restored, resulting in the target pose of the mobile robot and the target poses of each feature point in the target scene. This improves the accuracy of the determined target pose of the mobile robot and the target poses of each feature point in the target scene.

[0098] In one application scenario, generating a map of the target scene based on the target pose of each feature point in the target scene can improve the accuracy of the generated map.

[0099] In another application scenario, the accuracy of path planning can be improved by subsequently planning the mobile robot's movement path in the target scene based on the target pose of the mobile robot and the target pose of each feature point in the target scene.

[0100] See Figure 1 , Figure 1 A flowchart of a pose restoration method provided in this application embodiment is shown. The method is applied to an electronic device, which includes a camera and a sensor. The method may include the following steps:

[0101] S101: Acquire the target image of the target scene captured by the current camera, and the target depth data of the target scene captured by the current sensor.

[0102] The sensor is located within the camera's field of view; the target depth data includes the distance between the first feature point within the sensor's acquisition range in the target scene and the electronic device.

[0103] S102: Based on the target image and the previous frame image of the target image, determine the initial pose of the current electronic device and the initial pose of the second feature point within the camera's field of view in the target scene.

[0104] The initial pose of the electronic device is its three-dimensional coordinates in the world coordinate system; the initial pose of the second feature point is its three-dimensional coordinates in the world coordinate system.

[0105] S103: Based on the image region occupied by the first feature point in the target image, extract a reference image containing the third feature point from the target image.

[0106] Among them, the third feature point belongs to the first feature point.

[0107] S104: Calculate the scale restoration parameters of the target image based on the target depth data of the third feature point in the reference image and the initial pose of the current third feature point.

[0108] S105: According to the scale repair parameters, repair the initial pose of the current electronic device and the initial pose of the second feature point to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0109] Based on the pose restoration method provided in this application, the target depth data of the third feature point can represent the pose of the second feature point obtained based on the sensor, and the initial pose of the second feature point represents the pose of the third feature point obtained based on the camera. Correspondingly, the scale restoration parameter calculated based on the target depth data of the third feature point and the initial pose of the third feature point can represent the difference between the pose of the third feature point obtained based on the sensor and the pose of the third feature point obtained based on the camera. Furthermore, the scale of the pose change of the electronic device, the scale of the pose change of the second feature point, and the scale of the pose change of the third feature point are the same. Therefore, restoring the initial pose of the electronic device and the initial pose of the second feature point in the target scene based on the scale restoration parameter can improve the accuracy of the determined target pose of the electronic device and the target pose of the second feature point.

[0110] In some embodiments, the electronic device is a mobile robot; the mobile robot is equipped with a logic operation unit, which is used to execute the pose repair method provided in the embodiments of this application. The camera is a monocular camera; the optical axis of the camera is parallel to the horizontal plane; the sensor is a single-point TOF (Time of Flight) sensor; the sensor is located directly above or below the camera.

[0111] For example, see Figure 2 , Figure 2 This diagram illustrates a camera and sensor mounting structure according to an embodiment of this application. A monocular camera is mounted on a mobile robot, with its optical axis parallel to the horizontal plane. A single-point Time-of-Flight (TOF) sensor is mounted directly below the camera; the signal emission direction of the single-point TOF sensor can also be parallel to the horizontal plane.

[0112] Furthermore, since the sensor is located within the camera's FOV (Field of View), the target image captured by the camera in the target scene includes the image of the region where the sensor has collected the target depth data. This allows for the acquisition of both the image and depth data of the same region within the target scene. Subsequently, the pose of the mobile robot and the poses of various feature points in the target scene can be repaired based on the image and depth data of this region, improving the accuracy of the determined target pose of the mobile robot and the target poses of various feature points in the target scene.

[0113] Regarding step S101, the electronic device is a mobile robot. During its movement within the target scene, the mobile robot captures images of the target scene in real time using a camera. The target image is the image of the target scene captured by the mobile robot at the current moment. Furthermore, the electronic device also captures depth data of the target scene in real time using sensors. This target depth data includes the distance between the first feature point within the sensor's acquisition range in the target scene and the electronic device.

[0114] Regarding step S102, since the camera captures an image of the area within the camera's field of view in the target scene, the target image includes feature points (i.e., second feature points) within the camera's field of view in the target scene. Accordingly, the electronic device can perform SLAM based on the target image captured by the camera to obtain the initial pose of the current electronic device and the initial poses of each second feature point in the target scene.

[0115] In some embodiments, Figure 1 Based on this, see Figure 3 Step S102 may include the following steps:

[0116] S1021: For each second feature point in the target image, generate a feature descriptor for the second feature point based on the distance between the second feature point and other feature points within its neighborhood.

[0117] S1022: For each fourth feature point in the previous frame of the target image, generate a feature descriptor for the fourth feature point based on the distance between the fourth feature point and other feature points within the neighborhood of the fourth feature point.

[0118] S1023: Based on the feature descriptors of each second feature point and each fourth feature point, calculate the matching relationship between each second feature point and each fourth feature point.

[0119] S1024: Based on the matching relationship between each second feature point and each fourth feature point, calculate the transformation relationship from the previous frame image of the target image to the target image, and determine the initial pose of the current electronic device based on the transformation relationship.

[0120] S1025: Based on the two-dimensional coordinates of each second feature point in the target image and the camera parameters, calculate the three-dimensional coordinates of each second feature point in the world coordinate system to obtain the initial pose of the second feature points within the camera's field of view in the target scene.

[0121] For each feature point, the feature descriptor is ORB (Oriented Fast and Rotated BRIEF). The neighborhood of the feature point can be a region centered on the feature point with a specified radius; or, the neighborhood of the feature point can be a rectangular region centered on the feature point with a specified width and length.

[0122] For each second feature point in the target image, multiple feature points are randomly selected within the neighborhood of that second feature point. Based on the pixel values ​​of the selected multiple feature points, a feature descriptor for that second feature point is generated. For example, for every two selected feature points, binary encoding is performed with the feature point having a larger pixel value as 1 and the feature point having a smaller pixel value as 0 to obtain the feature descriptor for that second feature point.

[0123] The electronic device determines the feature descriptor of the fourth feature point in a similar way to determine the feature descriptor of the second feature point, and can refer to the relevant description in the foregoing embodiments.

[0124] For each second feature point in the target image, the electronic device selects a fourth feature point from the fourth feature points in the previous frame of the target image as the current feature point to be matched, and calculates the Hamming distance between the feature descriptor of the second feature point and the feature descriptor of the current feature point to be matched.

[0125] If the calculated Hamming distance is not less than the second threshold, then the second feature point is determined to match the current feature point to be matched. If the calculated Hamming distance is less than the second threshold, then the second feature point is determined not to match the current feature point to be matched. The electronic device selects one unmatched fourth feature point from the previous frame of the target image as the current feature point to be matched, and calculates the Hamming distance between the feature descriptor of the second feature point and the feature descriptor of the current feature point to be matched. By doing so, the matching fourth feature point in the previous frame of the target image for each second feature point can be determined, thus obtaining the matching relationship between each second feature point and each fourth feature point.

[0126] A second feature point in the target image matching a fourth feature point in the previous frame indicates that the second feature point and the fourth feature point are identical. Transforming the fourth feature point in the previous frame yields a matching second feature point in the target image.

[0127] The electronic device can calculate the transformation relationship from the previous frame image to the target image based on the two-dimensional coordinates of the second feature point in the target image and the two-dimensional coordinates of the fourth feature point that matches the second feature point in the previous frame image. The calculated transformation relationship includes the rotation matrix R and translation vector T from the previous frame image to the target image. That is, the fourth feature point in the previous frame image is transformed according to the calculated rotation matrix R and translation vector T to obtain the matching second feature point in the target image.

[0128] The electronic device transforms the pose of the electronic device corresponding to the previous frame image based on the calculated transformation relationship to obtain the initial pose of the current electronic device. The pose of the electronic device corresponding to the previous frame image can be determined according to the pose repair method provided in the embodiments of this application.

[0129] Furthermore, based on the two-dimensional coordinates of each second feature point in the target image and the camera parameters, the three-dimensional coordinates of each second feature point in the world coordinate system are calculated to obtain the initial pose of the second feature points within the camera's field of view in the target scene. The camera parameters include the camera's intrinsic parameters and extrinsic parameters.

[0130] The pose restoration method provided in this application can determine the initial pose of an electronic device and the initial pose of a second feature point in the target scene based on the target image acquired by the camera and the target depth data acquired by the sensor. Subsequently, the initial pose of the electronic device and the initial pose of the second feature point in the target scene can be restored based on scale restoration parameters, which can improve the accuracy of the determined target pose of the electronic device and the target pose of the second feature point.

[0131] Regarding step S103, since the sensor is located within the camera's field of view, the target image captured by the camera includes the image of the region where the sensor collects depth data. Correspondingly, the second feature point within the camera's field of view includes the first feature point within the sensor's collection range. Therefore, the electronic device can determine the image region occupied by the first feature point in the target image, which is the image of the region where the sensor collects depth data.

[0132] In one implementation, the electronic device can directly determine the image area occupied by the first feature point in the target image as a reference image. In this case, the first feature point is always the third feature point.

[0133] In another implementation, Figure 1 Based on this, see Figure 4 Step S103 may include the following steps:

[0134] S1031: Determine the image area occupied by each first feature point in the target image according to the pre-recorded location of the calibration area, and obtain the candidate image.

[0135] The location of the calibration area is determined by calibration based on the images to be processed pre-acquired by the camera.

[0136] S1032: Perform target detection on the target image to obtain the image region occupied by each target object in the target image, and obtain the object image.

[0137] S1033: Compare the candidate image with the object image. If the candidate image includes a target object, determine the image to be processed as the reference image.

[0138] S1034: If the candidate image includes multiple target objects, extract the image area occupied by the target object with the largest area from the candidate image to obtain the reference image.

[0139] The electronic device acquires the position of a pre-recorded calibration area. This pre-recorded calibration area, determined during camera and sensor calibration, represents the position of a feature point within the sensor's acquisition range within the image to be processed captured by the camera. Therefore, the electronic device, based on the pre-recorded calibration area position, determines the corresponding image area within the image to be processed, which is also the image area occupied by the first feature point within the sensor's acquisition range in the target image.

[0140] The electronic device performs target detection on the target image based on a preset algorithm, obtaining the image region occupied by each target object in the target image, and thus obtaining the object image. The preset algorithm can be Mask-RCNN (Mask Recycle Convolutional Neural Network).

[0141] Then, the electronic device compares the object image with the candidate image. For example, the electronic device determines the overlapping area between the object image and the candidate image according to the position of the object image in the image to be processed and the position of the candidate image in the image to be processed.

[0142] If the candidate image overlaps with only one object image, then the candidate image contains a target object, indicating that the target object is close to the electronic device and occupies a large area in the target image. Therefore, the accuracy of the target depth data acquired by the sensor for this target object is high. Thus, the electronic device can directly identify the candidate image as the reference image. Subsequently, the accuracy of the calculated scale restoration parameters can be improved, thereby enhancing the accuracy of the determined target pose and the target pose of the second feature point of the electronic device.

[0143] If the candidate image only overlaps with multiple object images, then the candidate image includes multiple target objects at varying distances from the electronic device. The electronic device can identify the largest target object in the candidate image, which is closer to the electronic device, resulting in higher accuracy of the target depth data acquired by the sensor. Subsequently, the accuracy of the calculated scale restoration parameters can be improved, thereby enhancing the accuracy of the determined target pose and second feature point pose of the electronic device. Therefore, the electronic device can directly determine the image region occupied by the largest target object in the candidate image, obtaining a reference image. In this case, the third feature point in the reference image is the feature point of the largest target object in the candidate image.

[0144] In some embodiments, if the candidate image does not contain the target object, it may be because the target object is far from the electronic device, or the target object occupies a small area in the target image, and therefore is not detected during target detection. When the target object is far from the electronic device, the accuracy of the target depth data acquired by the sensor is low. Therefore, the electronic device can directly determine the initial pose of the current electronic device and the initial pose of the second feature point within the camera's field of view in the target scene as the target pose without repairing the initial pose of the current electronic device and the initial pose of the second feature point.

[0145] In some embodiments, Figure 4 Based on this, see Figure 5 Before step S1031, the method may further include the following steps:

[0146] S106: Acquire the image to be processed from the camera, including the calibration board placed in the target scene, and the depth data of the calibration board acquired by the sensor.

[0147] The depth data of the calibration board includes the distance between each feature point on the calibration board and the electronic device.

[0148] S107: Extract the image region occupied by the calibration plate from the image to be processed to obtain the calibration image.

[0149] S108: Based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image and the camera parameters of the camera, determine the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system.

[0150] S109: Calculate the calibration parameters of the calibration image based on the distance between each feature point on the calibration board and the electronic device, and the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system.

[0151] S1010: If the calibration parameters are not less than the first threshold, determine a new calibration image from the image to be processed, and return to the step of performing the step of determining the three-dimensional coordinates of each feature point in the calibration board in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration board in the calibration image and the camera parameters of the camera.

[0152] S1011: If the calibration parameter is less than the first threshold, determine the position of the current calibration image in the image to be processed, and obtain the position of the calibration area.

[0153] The calibration board can be an Apriltag code panel, or it can be a checkerboard calibration board. The calibration board is placed vertically in the target scene at a preset distance (denoted as d1) from the electronic device. For example, place the calibration board vertically against a wall in the target scene, place the electronic device at a distance d1 from the calibration board, and have the patterned side of the calibration board facing the electronic device.

[0154] The camera in the electronic device can acquire an image to be processed, including a calibration board placed in the target scene, and the sensor can acquire depth data of the calibration board. The electronic device performs target detection on the image to be processed, determines the image area occupied by the calibration board in the image to be processed, and obtains a calibration image. Then, based on the two-dimensional coordinates of each feature point in the calibration board in the calibration image, and the camera parameters, the electronic device can determine the three-dimensional coordinates of each feature point in the calibration board in the world coordinate system.

[0155] The depth data of the calibration board includes the distances between each feature point on the calibration board and the electronic device. The three-dimensional coordinates of each feature point on the calibration board in the world coordinate system can also represent the distances between the feature points on the calibration board and the electronic device. Ideally, the distances between the feature points on the calibration board and the electronic device obtained based on sensors are the same as the distances obtained based on cameras.

[0156] Therefore, the electronic device can calculate the calibration parameters of the calibration image based on the distance between each feature point on the calibration board and the electronic device, as well as the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system.

[0157] In some embodiments, step S107 includes the following steps: calculating calibration parameters of the calibration image based on the distance between each feature point in the calibration board and the electronic device, the three-dimensional coordinates of each feature point in the world coordinate system and a first preset formula.

[0158] The first preset formula is:

[0159]

[0160] Δd represents the calibration parameters of the calibration image; The mean distance between each feature point on the calibration board and the electronic device is represented by z; N represents the number of feature points on the calibration board; i This represents the coordinate value of the i-th feature point in the calibration plate on a specified coordinate axis in the world coordinate system.

[0161] The X and Y axes of the world coordinate system lie in the horizontal plane, and the Z axis is vertical. The Z axis of the world coordinate system is specified. The coordinates of a feature point on the calibration plate along the Z axis of the world coordinate system represent the distance between the feature point on the calibration plate and the electronic device.

[0162] For each feature point on the calibration board, the distance between that feature point and the electronic device is denoted as D. All feature points on the calibration board lie on the same plane. Ideally, the distances between all feature points on the calibration board and the electronic device should be the same. However, due to sensor errors, the distances between the feature points on the calibration board and the electronic device obtained from the sensor differ. Therefore, the electronic device calculates the average distance between all feature points on the calibration board and the electronic device to obtain...

[0163] Similarly, due to camera errors, the distances between the feature points on the calibration board obtained from the camera and the electronic device vary. Therefore, the electronic device calculates the average distance between each feature point on the calibration board and the electronic device, which is essentially calculating the average coordinate values ​​of each feature point on the calibration board along a specified coordinate axis in the world coordinate system.

[0164] Due to sensor and camera errors, the distances between feature points on the calibration board and the electronic device obtained based on the sensor differ from those obtained based on the camera. The calibration parameters of the calibration image can represent these differences.

[0165] If the calibration parameter is less than the first threshold, it indicates that there is a small difference between the distances between the feature points in the calibration board obtained from the sensor and the electronic device, and the distances between the feature points in the calibration board obtained from the camera and the electronic device. Therefore, the determined calibration image includes the feature points of the calibration board. Thus, the electronic device can determine the position of the current calibration image in the image to be processed, and obtain the position of the calibration region.

[0166] If the calibration parameters are not less than the first threshold, it indicates a significant difference between the distances between the feature points on the calibration board obtained from the sensor and the electronic device, and the distances between the feature points on the calibration board obtained from the camera and the electronic device. This may be because the determined calibration image includes feature points other than those on the calibration board. Therefore, the electronic device can determine a new calibration image from the image to be processed.

[0167] In some embodiments, step S108, which involves determining a new calibration image from the image to be processed, includes: offsetting the previously determined calibration image in the image to be processed by a preset offset to obtain a new calibration image.

[0168] In one implementation, the electronic device calculates the sum of the two-dimensional coordinates of the center point of the previously determined calibration image in the image to be processed and a preset offset, to obtain the two-dimensional coordinates of the center point of the offset calibration image in the image to be processed, and determines an image region in the image to be processed with the same width and length as the previously determined calibration image according to the center point of the offset calibration image, to obtain a new calibration image.

[0169] In another implementation, for each vertex of the calibration image determined in this instance, the electronic device obtains the offset two-dimensional coordinates of the vertex in the image to be processed by summing the two-dimensional coordinates of the vertex in the image to be processed with a preset offset, and obtains a new calibration image according to the offset two-dimensional coordinates of each vertex in the image to be processed.

[0170] Then, the electronic device can determine the three-dimensional coordinates of each feature point in the calibration board in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration board in the determined calibration image, and the camera parameters of the camera, and calculate the calibration parameters of the determined calibration image. This process is repeated until the calibration parameters of the determined calibration image are less than a first threshold. The electronic device then determines the position of the current calibration image in the image to be processed, thus obtaining the position of the calibration region. The position of the calibration region can be represented by the two-dimensional coordinates of the four vertices of the calibration region in the image to be processed.

[0171] The pose restoration method provided in this application can be used to calibrate the camera and sensor to obtain the position of the calibration area. Then, a reference image can be determined based on the position of the calibration area, and the efficiency of the reference image can be determined, thereby improving the efficiency of pose restoration.

[0172] In some embodiments, to improve the accuracy of the determined calibration area, the distance d1 between the electronic device and the calibration plate can be set to a set of distance values, such as 1 meter, 1.5 meters, and 2 meters. Accordingly, the electronic device is placed at a distance of 1 meter from the calibration plate, and the camera and sensor are calibrated as described above. Then, the electronic device is placed at a distance of 1.5 meters from the calibration plate, and the camera and sensor are calibrated as described above. Finally, the electronic device is placed at a distance of 2 meters from the calibration plate, and the camera and sensor are calibrated as described above.

[0173] In some embodiments, the electronic device can also save the position of the calibration region. Subsequently, if the mounting positions of the camera and sensor in the electronic device remain unchanged, when determining the pose of the electronic device, the saved position of the calibration region can be directly obtained, and a reference image can be determined based on the obtained position of the calibration region, which can improve the pose repair efficiency. If the mounting positions of the camera or sensor in the electronic device are changed, when determining the pose of the electronic device, the position of the calibration region can be determined in the manner described above, and a reference image can be determined based on the obtained position of the calibration region.

[0174] Regarding step S104, the target depth data of the third feature point can represent the pose of the second feature point obtained from the sensor, and the initial pose of the second feature point represents the pose of the third feature point obtained from the camera. Therefore, the electronic device can calculate the scale restoration parameters of the target image based on the target depth data of the third feature point in the reference image and the initial pose of the current third feature point. The calculated scale restoration parameters can represent the difference between the pose of the third feature point obtained from the sensor and the pose of the third feature point obtained from the camera.

[0175] In some embodiments, Figure 1 Based on this, see Figure 6 Step S104 may include the following steps:

[0176] S1041: Calculate the mean distance between the third feature point and the electronic device to obtain the first mean distance.

[0177] S1042: Calculate the mean of the coordinates of the third feature point on the specified coordinate axes of the world coordinate system, and use it as the second distance mean.

[0178] S1043: Calculate the ratio of the first distance mean to the second distance mean to obtain the scale restoration parameters of the target image.

[0179] The coordinates of the third feature point on the specified coordinate axis of the world coordinate system are: the coordinates of the third feature point on the Z-axis of the world coordinate system, which represent the distance between the third feature point and the electronic device.

[0180] For each third feature point in the reference image, the distance between the third feature and the electronic device is denoted as D. Tof Calculate the first mean distance between each feature point and the electronic device, denoted as . The coordinate value of this third feature on the Z-axis of the world coordinate system is denoted as D. ROI Calculate the first mean distance between each feature point and the electronic device, denoted as .

[0181] Furthermore, the electronic device calculates the ratio of the first distance mean to the second distance mean based on the following formula (2) to obtain the scale repair parameters of the target image.

[0182]

[0183] S repair This represents the scale restoration parameters of the target image. This represents the mean of the first distance. This represents the mean of the second distance.

[0184] In some embodiments, before calculating the scale restoration parameters of the target image, the electronic device may also determine whether preset scale restoration conditions are met. When the electronic device performs target detection on the target image, it obtains the object image of the target object in the target image, denoted as OB = {ob i Let ,i=1,2,3…}, where OB is the set of all target objects. i Let be the object image of the i-th target object.

[0185] After determining the candidate images occupied by the first feature point in the target image, the candidate images are compared with the object image. If there is an overlapping area between the candidate image and an object image, the candidate image is determined to contain the target object in that object image. The number of target objects contained in the candidate image is denoted as cnt. If cnt > 0, it means that the candidate image contains at least one target object and that the candidate image contains the background. If cnt = 1, Then the scale repair condition is satisfied. Here, ζ is a preset distance threshold. If cnt > 1, and This confirms that the scale restoration conditions are met. Subsequently, the scale restoration parameters S of the target image are calculated using the method described above. repair.

[0186] If cnt < 1, it indicates that the candidate images do not contain the target object; or, This indicates that the distance between the target object and the electronic device is relatively large, so the accuracy of scale repair based on the distance between the target object and the electronic device is low. Therefore, it can be determined that the scale repair conditions are not met, and no further processing is performed.

[0187] For step S105, the electronic device calculates the product of the initial pose of the current electronic device and the scale repair parameters to obtain the target pose of the current electronic device, and calculates the product of the initial pose of the second feature point and the scale repair parameters to obtain the target pose of the second feature point.

[0188] The target pose of the electronic device is: the three-dimensional coordinates of the electronic device in the world coordinate system at the time the target image is acquired. The target pose of the second feature point is: the three-dimensional group of the second feature point within the camera's field of view in the world coordinate system at the time the target image is acquired.

[0189] Correspondingly, when the electronic device is a mobile robot, during the movement of the electronic device in the target scene, the pose of the electronic device can be determined in real time based on the images collected by the electronic device in real time and the depth data collected by the sensors in real time, and the pose of each feature point in the target scene can be obtained.

[0190] See Figure 7 , Figure 7 This is a flowchart illustrating a pose restoration method provided in an embodiment of this application. The method is applied to a mobile robot, which includes a monocular camera and a single-point Time-of-Flight (TOF) sensor. The method may include the following steps:

[0191] S701: Acquire monocular image data.

[0192] In this step, the monocular image data is the image captured by the monocular camera, which is the image to be processed including the calibration board in the target scene in the aforementioned embodiment.

[0193] S702: Acquire single-point TOF (Time of Flight) data.

[0194] In this step, the single-point TOF data is the depth data of the calibration board collected by the single-point TOF sensor.

[0195] S703: Image and single-point TOF corresponding ROI (Region of Interest) calibration.

[0196] In this step, the area measured by the single-point TOF sensor and the image are calibrated to obtain the corresponding Region of Interest (ROI). The ROI is the calibration image in the aforementioned embodiment. In other words, the mobile robot calibrates the image to be processed, which includes the calibration board, based on the image acquired by the monocular camera and the depth data of the calibration board acquired by the single-point TOF sensor, to determine the calibration image.

[0197] S704: Determine whether the calibration was successful. If yes, proceed to step S705; otherwise, proceed to step S703.

[0198] In this step, the mobile robot calculates the calibration parameters of the calibration image. If the calibration parameters are less than a first threshold, the calibration is successful, and the mobile robot determines the position of the current calibration image in the image to be processed, thus obtaining the position of the calibration region. If the calibration parameters are not less than the first threshold, the calibration fails. The mobile robot can then perform calibration again based on the image to be processed containing the calibration board acquired by the monocular camera and the depth data of the calibration board acquired by the single-point TOF sensor, obtaining a new calibration image. This process continues until the calibration parameters of the determined calibration image are less than the first threshold. The mobile robot then determines the position of the current calibration image in the image to be processed, thus obtaining the position of the calibration region.

[0199] S705: Determine whether the monocular initialization was successful. If yes, execute steps S706 and S708. If no, execute step S701.

[0200] In this step, successful monocular initialization means determining the robot's pose at startup based on the first frame image captured by the monocular camera. If monocular initialization fails, meaning the robot's pose at startup is not determined, the robot can undergo monocular initialization again until its initial pose is determined.

[0201] S706: Adjacent frame matching.

[0202] In this step, adjacent frames refer to the target image acquired by the mobile robot and the previous frame image of the target image. Adjacent frame matching means that the mobile robot matches each second feature point in the target image with the fourth feature point in the previous frame image to obtain the matching relationship between each second feature point and each fourth feature point.

[0203] S707: Camera pose and map point pose estimation.

[0204] In this step, since the camera is mounted on the mobile robot, its pose is the same as the robot's pose. The camera pose is the initial pose of the mobile robot, and the map points are the second feature points within the camera's field of view in the target scene. Based on the matching relationship between each second feature point and a fourth feature point, the mobile robot determines its initial pose. Furthermore, based on the two-dimensional coordinates of each second feature point in the target image and the camera parameters, the initial pose of each second feature point is calculated.

[0205] S708: Target segmentation.

[0206] In this step, the mobile robot performs target segmentation on the target image to obtain object images of each target object in the target image.

[0207] S709: ROI region deep recovery.

[0208] In this step, the ROI region is the candidate image in the aforementioned embodiments. Based on the pre-recorded location of the calibration region, the mobile robot determines the image region occupied by each first feature point in the target image, thus obtaining the candidate image.

[0209] S7010: Determine if the depth of the ROI region is uniform. If so, proceed to step S7011.

[0210] In this step, determining whether the depth of the ROI region is uniform means determining how many target objects are contained in the candidate image. If the candidate image contains a target object and the target depth data of each third feature point of the target object are the same, then the depth of the ROI region is uniform, and the mobile robot can determine the candidate image as the reference image.

[0211] If the candidate image contains multiple target objects and the target depth data of each third feature point of the multiple target objects are different, then the depth of the ROI region is not uniform. The mobile robot extracts the image region occupied by the target object with the largest area from the candidate image to obtain the reference image.

[0212] S7011: Repair camera pose and map point pose.

[0213] In this step, since the camera is mounted on the mobile robot, the camera pose is the same as the mobile robot's pose. The camera pose is also the initial pose of the mobile robot, and the map points are the second feature points within the camera's field of view in the target scene.

[0214] Based on the target depth data and initial pose of the third feature point in the reference image, the mobile robot calculates the scale restoration parameters of the target image, and restores the initial pose of the mobile robot and the initial pose of the second feature point in the target scene according to the scale restoration parameters, thereby obtaining the target pose of the mobile robot and the target pose of the second feature point.

[0215] Based on the pose restoration method provided in this application, the image region (i.e., calibration region) occupied by the single-point TOF sensor in the image acquired by the camera can be calibrated in the early stage to find the ROI region corresponding to the depth measurement data of the single-point TOF sensor. Then, during the feature tracking and matching process of the target image and the previous frame image, the depth of the feature points in the ROI region is estimated, that is, the initial pose of the mobile robot and the initial position of the feature points in the target scene are determined. Furthermore, the target depth data measured by the single-point TOF sensor is compared with the initial pose obtained based on the target image acquired by the camera to obtain the scale ratio between the target depth data and the initial pose (i.e., the scale restoration parameter). Finally, the initial pose of the mobile robot and the initial pose of the map points (i.e., the feature points in the target image) are calibrated based on the calculated scale restoration parameter, which can improve the localization accuracy of the algorithm. Moreover, the pose restoration method provided in this application is relatively simple in principle and implementation, and can be quickly applied to various indoor mobile robot products. It has the characteristics of strong adaptability, low cost, small code volume, and fast deployment, and has a wide range of application scenarios.

[0216] and Figure 1 For the corresponding method implementation examples, see [link to relevant documentation]. Figure 8 , Figure 8 This is a structural diagram of a pose restoration device applied to an electronic device, the electronic device including a camera and a sensor, the device comprising:

[0217] The first acquisition module 801 is used to acquire the target image of the target scene currently captured by the camera, and the target depth data of the target scene currently captured by the sensor; wherein the sensor is located within the field of view of the camera; the target depth data includes the distance between a first feature point within the sensor's acquisition range in the target scene and the electronic device;

[0218] The initial pose determination module 802 is used to determine the initial pose of the current electronic device and the initial pose of the second feature point within the camera's field of view in the target scene based on the target image and the previous frame image of the target image; wherein, the initial pose of the electronic device is the three-dimensional coordinate of the electronic device in the world coordinate system; and the initial pose of the second feature point is the three-dimensional coordinate of the second feature point in the world coordinate system.

[0219] The reference image acquisition module 803 is used to extract a reference image containing a third feature point from the target image based on the image area occupied by the first feature point in the target image; wherein the third feature point belongs to the first feature point;

[0220] The scale restoration parameter determination module 804 is used to calculate the scale restoration parameters of the target image based on the target depth data of the third feature point in the reference image and the current initial pose of the third feature point.

[0221] The target pose determination module 805 is used to repair the initial pose of the current electronic device and the initial pose of the second feature point according to the scale repair parameters, so as to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0222] Optionally, the scale repair parameter determination module 804 is specifically used to calculate the average distance between the third feature point and the electronic device to obtain a first average distance.

[0223] Calculate the mean of the coordinate values ​​of the third feature point on a specified coordinate axis in the world coordinate system, and use it as the second distance mean;

[0224] The ratio of the first mean distance to the second mean distance is calculated to obtain the scale restoration parameters of the target image.

[0225] Optionally, the target pose determination module 805 is specifically used to calculate the product of the initial pose of the current electronic device and the scale repair parameter, and the product of the initial pose of the second feature point and the scale repair parameter, respectively, to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0226] Optionally, the reference image acquisition module 803 is specifically used to determine the image area occupied by each first feature point in the target image according to the position of the pre-recorded calibration area, so as to obtain a candidate image; wherein, the position of the calibration area is determined by calibration based on the image to be processed pre-acquired by the camera;

[0227] Target detection is performed on the target image to obtain the image region occupied by each target object in the target image, thus obtaining the object image;

[0228] The candidate image is compared with the object image. If the candidate image contains a target object, the image to be processed is determined as the reference image.

[0229] If the candidate image includes multiple target objects, the image region occupied by the target object with the largest area is extracted from the candidate image to obtain a reference image.

[0230] Optionally, the device further includes:

[0231] The second acquisition module is used to acquire, before the reference image acquisition module 803 performs the extraction of a reference image containing a third feature point from the target image based on the image region occupied by the first feature point in the target image, an image to be processed including a calibration board placed in the target scene captured by the camera, and depth data of the calibration board captured by the sensor; wherein, the depth data of the calibration board includes the distance between each feature point in the calibration board and the electronic device;

[0232] The calibration image acquisition module is used to extract the image region occupied by the calibration plate from the image to be processed, and obtain the calibration image;

[0233] The three-dimensional coordinate determination module is used to determine the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image and the camera parameters of the camera.

[0234] The calibration parameter determination module is used to calculate the calibration parameters of the calibration image based on the distance between each feature point in the calibration board and the electronic device, and the three-dimensional coordinates of each feature point in the world coordinate system.

[0235] The calibration image update module is used to determine a new calibration image from the image to be processed if the calibration parameter is not less than a first threshold, and to trigger the three-dimensional coordinate determination module to perform the step of determining the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image and the camera parameters of the camera.

[0236] The calibration region determination module is used to determine the position of the current calibration image in the image to be processed if the calibration parameter is less than the first threshold, thereby obtaining the position of the calibration region.

[0237] Optionally, the calibration image update module is specifically used to offset the previously determined calibration image in the image to be processed according to a preset offset to obtain a new calibration image.

[0238] Optionally, the calibration parameter determination module is specifically used to calculate the calibration parameters of the calibration image based on the distance between each feature point on the calibration board and the electronic device, the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system, and a first preset formula; wherein, the first preset formula is:

[0239]

[0240] Δd represents the calibration parameters of the calibration image; The z represents the average distance between each feature point in the calibration board and the electronic device; N represents the number of feature points in the calibration board; z i This represents the coordinate value of the i-th feature point in the calibration plate on a specified coordinate axis in the world coordinate system.

[0241] Optionally, the initial pose determination module 802 is specifically used to generate a feature descriptor for each second feature point in the target image based on the distance between the second feature point and other feature points within the neighborhood of the second feature point.

[0242] For each fourth feature point in the previous frame of the target image, a feature descriptor for the fourth feature point is generated based on the distance between the fourth feature point and other feature points within the neighborhood of the fourth feature point.

[0243] Based on the feature descriptors of each second feature point and each fourth feature point, calculate the matching relationship between each second feature point and each fourth feature point;

[0244] Based on the matching relationship between each second feature point and each fourth feature point, the transformation relationship from the previous frame image of the target image to the target image is calculated, and the initial pose of the current electronic device is determined based on the transformation relationship.

[0245] Based on the two-dimensional coordinates of each second feature point in the target image and the camera parameters of the camera, the three-dimensional coordinates of each second feature point in the world coordinate system are calculated to obtain the initial pose of the second feature points within the field of view of the camera in the target scene.

[0246] Optionally, the electronic device is a mobile robot; the camera is a monocular camera; the optical axis of the camera is parallel to the horizontal plane; the sensor is a single-point TOF sensor; and the sensor is located directly above or below the camera.

[0247] Based on the pose restoration apparatus provided in this application embodiment, the target depth data of the third feature point can represent the pose of the second feature point obtained based on the sensor, and the initial pose of the second feature point represents the pose of the third feature point obtained based on the camera. Correspondingly, the scale restoration parameters calculated based on the target depth data of the third feature point and the initial pose of the third feature point can represent the difference between the pose of the third feature point obtained based on the sensor and the pose of the third feature point obtained based on the camera. Furthermore, the scale of the pose change of the electronic device, the scale of the pose change of the second feature point, and the scale of the pose change of the third feature point are the same. Therefore, restoring the initial pose of the electronic device and the initial pose of the second feature point in the target scene based on the scale restoration parameters can improve the accuracy of the determined target pose of the electronic device and the target pose of the second feature point.

[0248] This application also provides an electronic device, such as... Figure 9 As shown, it includes:

[0249] Memory 901 is used to store computer programs;

[0250] When processor 902 executes a program stored in memory 901, it performs the following steps:

[0251] The system acquires a target image of the target scene captured by the current camera, and target depth data of the target scene captured by the current sensor; wherein the sensor is located within the field of view of the camera; the target depth data includes the distance between a first feature point within the sensor's acquisition range in the target scene and the electronic device.

[0252] Based on the target image and the previous frame image of the target image, the initial pose of the current electronic device and the initial pose of the second feature point within the camera's field of view in the target scene are determined; wherein, the initial pose of the electronic device is the three-dimensional coordinate of the electronic device in the world coordinate system; the initial pose of the second feature point is the three-dimensional coordinate of the second feature point in the world coordinate system;

[0253] Based on the image region occupied by the first feature point in the target image, a reference image containing a third feature point is extracted from the target image; wherein, the third feature point belongs to the first feature point;

[0254] Based on the target depth data of the third feature point in the reference image and the current initial pose of the third feature point, the scale restoration parameters of the target image are calculated.

[0255] According to the scale repair parameters, the initial pose of the current electronic device and the initial pose of the second feature point are repaired to obtain the target pose of the current electronic device and the target pose of the second feature point.

[0256] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 902, communication interface, and memory 901 communicating with each other via the communication bus.

[0257] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0258] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0259] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0260] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0261] In another embodiment provided in this application, a mobile robot system is also provided, including a data acquisition module and a processor; the data acquisition module includes a camera and sensors;

[0262] The data acquisition module is used to acquire target images of the target scene through the camera, and to acquire target depth data of the target scene through the sensor;

[0263] The processor is used to execute any of the pose repair method steps described above.

[0264] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described pose repair methods.

[0265] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the pose repair methods described above.

[0266] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0267] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0268] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of devices, electronic devices, mobile robot systems, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0269] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A pose restoration method, characterized in that, The method is applied to an electronic device, the electronic device including a camera and a sensor, and the method includes: The system acquires a target image of the target scene captured by the camera and target depth data of the target scene captured by the sensor; wherein the sensor is located within the field of view of the camera; the target depth data includes the distance between a first feature point within the sensor's acquisition range in the target scene and the electronic device. Based on the target image and the previous frame image of the target image, the initial pose of the current electronic device and the initial pose of the second feature point within the camera's field of view in the target scene are determined; wherein, the initial pose of the electronic device is the three-dimensional coordinate of the electronic device in the world coordinate system; the initial pose of the second feature point is the three-dimensional coordinate of the second feature point in the world coordinate system; Based on the image region occupied by the first feature point in the target image, a reference image containing a third feature point is extracted from the target image; wherein, the third feature point belongs to the first feature point; Based on the target depth data of the third feature point in the reference image and the current initial pose of the third feature point, the scale restoration parameters of the target image are calculated. According to the scale repair parameters, the initial pose of the current electronic device and the initial pose of the second feature point are repaired to obtain the target pose of the current electronic device and the target pose of the second feature point. The step of extracting a reference image containing a third feature point from the target image based on the image region occupied by the first feature point in the target image includes: According to the pre-recorded location of the calibration area, the image area occupied by each first feature point in the target image is determined to obtain candidate images; wherein, the location of the calibration area is determined by calibration based on the image to be processed pre-acquired by the camera; Target detection is performed on the target image to obtain the image region occupied by each target object in the target image, thus obtaining the object image; The candidate image is compared with the object image. If the candidate image includes a target object, the candidate image is determined as the reference image. If the candidate image includes multiple target objects, the image region occupied by the target object with the largest area is extracted from the candidate image to obtain a reference image.

2. The method according to claim 1, characterized in that, The calculation of scale restoration parameters for the target image based on the target depth data of the third feature point in the reference image and the current initial pose of the third feature point includes: Calculate the average distance between the third feature point and the electronic device to obtain the first average distance; Calculate the mean of the coordinate values ​​of the third feature point on a specified coordinate axis in the world coordinate system, and use it as the second distance mean; The ratio of the first mean distance to the second mean distance is calculated to obtain the scale restoration parameters of the target image.

3. The method according to claim 1, characterized in that, The step of repairing the initial pose of the current electronic device and the initial pose of the second feature point according to the scale repair parameters to obtain the target pose of the current electronic device and the target pose of the second feature point includes: The product of the initial pose of the current electronic device and the scale repair parameter, and the product of the initial pose of the second feature point and the scale repair parameter are calculated respectively to obtain the target pose of the current electronic device and the target pose of the second feature point.

4. The method according to claim 1, characterized in that, Before extracting a reference image containing a third feature point from the target image based on the image region occupied by the first feature point in the target image, the method further includes: The system acquires an image of the calibration board placed in the target scene, captured by the camera, and depth data of the calibration board, captured by the sensor; wherein the depth data of the calibration board includes the distance between each feature point in the calibration board and the electronic device. The image region occupied by the calibration plate is extracted from the image to be processed to obtain the calibration image; Based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image, and the camera parameters of the camera, the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system are determined. Based on the distance between each feature point in the calibration board and the electronic device, and the three-dimensional coordinates of each feature point in the world coordinate system, the calibration parameters of the calibration image are calculated. If the calibration parameter is not less than the first threshold, a new calibration image is determined from the image to be processed, and the process returns to the step of determining the three-dimensional coordinates of each feature point in the calibration board in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration board in the calibration image and the camera parameters of the camera. If the calibration parameter is less than the first threshold, the position of the current calibration image in the image to be processed is determined, and the position of the calibration region is obtained.

5. The method according to claim 4, characterized in that, The step of determining a new calibration image from the image to be processed includes: In the image to be processed, the previously determined calibration image is offset according to a preset offset to obtain a new calibration image.

6. The method according to claim 4, characterized in that, The calculation of calibration parameters for the calibration image based on the distance between each feature point on the calibration board and the electronic device, and the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system, includes: Based on the distance between each feature point on the calibration board and the electronic device, and the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system, and a first preset formula, the calibration parameters of the calibration image are calculated; wherein, the first preset formula is: ; The calibration parameters represent the calibration image; The distance between each feature point in the calibration board and the electronic device is represented by N; N represents the number of feature points in the calibration board. This represents the coordinate value of the i-th feature point in the calibration plate on a specified coordinate axis in the world coordinate system.

7. The method according to claim 1, characterized in that, The step of determining the initial pose of the current electronic device and the initial pose of the second feature point within the camera's field of view in the target scene based on the target image and the previous frame image of the target image includes: For each second feature point in the target image, a feature descriptor for the second feature point is generated based on the distance between the second feature point and other feature points within its neighborhood. For each fourth feature point in the previous frame of the target image, a feature descriptor for the fourth feature point is generated based on the distance between the fourth feature point and other feature points within the neighborhood of the fourth feature point. Based on the feature descriptors of each second feature point and each fourth feature point, calculate the matching relationship between each second feature point and each fourth feature point; Based on the matching relationship between each second feature point and each fourth feature point, the transformation relationship from the previous frame image of the target image to the target image is calculated, and the initial pose of the current electronic device is determined based on the transformation relationship. Based on the two-dimensional coordinates of each second feature point in the target image and the camera parameters of the camera, the three-dimensional coordinates of each second feature point in the world coordinate system are calculated to obtain the initial pose of the second feature points within the field of view of the camera in the target scene.

8. The method according to any one of claims 1 to 7, characterized in that, The electronic device is a mobile robot; the camera is a monocular camera; the optical axis of the camera is parallel to the horizontal plane; the sensor is a single-point TOF sensor; the sensor is located directly above or below the camera.

9. A posture restoration device, characterized in that, The device is used in an electronic device, the electronic device including a camera and a sensor, and the device includes: The first acquisition module is used to acquire the target image of the target scene currently captured by the camera, and the target depth data of the target scene currently captured by the sensor; wherein the sensor is located within the field of view of the camera; the target depth data includes the distance between a first feature point within the sensor's acquisition range in the target scene and the electronic device; An initial pose determination module is used to determine the initial pose of the current electronic device and the initial pose of a second feature point within the camera's field of view in the target scene based on the target image and the previous frame image of the target image; wherein, the initial pose of the electronic device is the three-dimensional coordinates of the electronic device in the world coordinate system; and the initial pose of the second feature point is the three-dimensional coordinates of the second feature point in the world coordinate system. A reference image acquisition module is used to extract a reference image containing a third feature point from the target image based on the image area occupied by the first feature point in the target image; wherein the third feature point belongs to the first feature point; The scale restoration parameter determination module is used to calculate the scale restoration parameters of the target image based on the target depth data of the third feature point in the reference image and the current initial pose of the third feature point. The target pose determination module is used to repair the initial pose of the current electronic device and the initial pose of the second feature point according to the scale repair parameters, so as to obtain the target pose of the current electronic device and the target pose of the second feature point. The reference image acquisition module is specifically used to determine the image area occupied by each first feature point in the target image according to the position of the pre-recorded calibration area, so as to obtain a candidate image; wherein, the position of the calibration area is determined by calibration based on the image to be processed pre-acquired by the camera; Target detection is performed on the target image to obtain the image region occupied by each target object in the target image, thus obtaining the object image; The candidate image is compared with the object image. If the candidate image includes a target object, the candidate image is determined as the reference image. If the candidate image includes multiple target objects, the image region occupied by the target object with the largest area is extracted from the candidate image to obtain a reference image.

10. The apparatus according to claim 9, characterized in that, The scale repair parameter determination module is specifically used to calculate the mean distance between the third feature point and the electronic device to obtain the first mean distance. Calculate the mean of the coordinate values ​​of the third feature point on a specified coordinate axis in the world coordinate system, and use it as the second distance mean; Calculate the ratio of the first mean distance to the second mean distance to obtain the scale restoration parameters of the target image; The target pose determination module is specifically used to calculate the product of the initial pose of the current electronic device and the scale repair parameter, and the product of the initial pose of the second feature point and the scale repair parameter, respectively, to obtain the target pose of the current electronic device and the target pose of the second feature point. The device further includes: The second acquisition module is used to acquire, before the reference image acquisition module performs the extraction of a reference image containing a third feature point from the target image based on the image region occupied by the first feature point in the target image, an image to be processed including a calibration board placed in the target scene captured by the camera, and depth data of the calibration board captured by the sensor; wherein, the depth data of the calibration board includes the distance between each feature point in the calibration board and the electronic device. The calibration image acquisition module is used to extract the image region occupied by the calibration plate from the image to be processed, and obtain the calibration image; The three-dimensional coordinate determination module is used to determine the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image and the camera parameters of the camera. The calibration parameter determination module is used to calculate the calibration parameters of the calibration image based on the distance between each feature point in the calibration board and the electronic device, and the three-dimensional coordinates of each feature point in the world coordinate system. The calibration image update module is used to determine a new calibration image from the image to be processed if the calibration parameter is not less than a first threshold, and to trigger the three-dimensional coordinate determination module to perform the step of determining the three-dimensional coordinates of each feature point in the calibration plate in the world coordinate system based on the two-dimensional coordinates of each feature point in the calibration plate in the calibration image and the camera parameters of the camera. The calibration region determination module is used to determine the position of the current calibration image in the image to be processed if the calibration parameter is less than the first threshold, thereby obtaining the position of the calibration region. The calibration image update module is specifically used to offset the previously determined calibration image in the image to be processed by a preset offset to obtain a new calibration image. The calibration parameter determination module is specifically used to calculate the calibration parameters of the calibration image based on the distance between each feature point on the calibration board and the electronic device, the three-dimensional coordinates of each feature point on the calibration board in the world coordinate system, and a first preset formula; wherein, the first preset formula is: ; The calibration parameters represent the calibration image; The distance between each feature point in the calibration board and the electronic device is represented by N; N represents the number of feature points in the calibration board. This represents the coordinate value of the i-th feature point in the calibration plate on a specified coordinate axis in the world coordinate system; The initial pose determination module is specifically used to generate a feature descriptor for each second feature point in the target image based on the distance between the second feature point and other feature points within the neighborhood of the second feature point. For each fourth feature point in the previous frame of the target image, a feature descriptor for the fourth feature point is generated based on the distance between the fourth feature point and other feature points within the neighborhood of the fourth feature point. Based on the feature descriptors of each second feature point and each fourth feature point, calculate the matching relationship between each second feature point and each fourth feature point; Based on the matching relationship between each second feature point and each fourth feature point, the transformation relationship from the previous frame image of the target image to the target image is calculated, and the initial pose of the current electronic device is determined based on the transformation relationship. Based on the two-dimensional coordinates of each second feature point in the target image and the camera parameters of the camera, the three-dimensional coordinates of each second feature point in the world coordinate system are calculated to obtain the initial pose of the second feature points within the field of view of the camera in the target scene. The electronic device is a mobile robot; the camera is a monocular camera; the optical axis of the camera is parallel to the horizontal plane; the sensor is a single-point TOF sensor; the sensor is located directly above or below the camera.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

12. A mobile robot system, characterized in that, It includes a data acquisition module and a processor; the data acquisition module includes a camera and a sensor; The data acquisition module is used to acquire target images of the target scene through the camera, and to acquire target depth data of the target scene through the sensor; The processor is configured to perform the steps of the method according to any one of claims 1-8.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images