A cloud positioning method, device, apparatus and storage medium

Abstract

Embodiments of the present disclosure provide a cloud positioning method, device, equipment and storage medium, wherein the method comprises: obtaining a to-be-positioned image frame in a video; based on a database, performing pose solving on the to-be-positioned image frame; based on a result of the pose solving, determining whether the to-be-positioned image frame is a solving error image frame; when it is determined that the to-be-positioned image frame is a solving error image frame, discarding the result of the pose solving; and when it is determined that the to-be-positioned image frame is not a solving error image frame, positioning the to-be-positioned image frame based on the result of the pose solving.

Description

Technical Field

[0001] This disclosure relates to the field of positioning technology, and in particular to a cloud positioning method, apparatus, device and storage medium. Background Technology

[0002] Visual SLAM (Simultaneous Localization and Mapping) is a crucial technology for autonomous mobile devices equipped with visual sensors such as cameras to autonomously locate themselves and build a 3D map of their surroundings without any prior environmental information. It is a fundamental technology in fields such as computer vision, robotics, autonomous vehicles, and augmented reality. Based on visual SLAM technology, in unknown environments, it solves for the pose and 3D spatial map between camera image frames by tracking and extracting features from the image frames.

[0003] Existing device-side and cloud-based positioning fusion technologies perform device-side positioning and cloud-side positioning separately, then fuse the positioning information on the device to obtain the final positioning information. Because the cloud positioning results are directly sent to the device without evaluation, errors in the cloud-based visual positioning information can easily be introduced, causing visual jitter in the fused result, if the cloud-based visual positioning information is not properly resolved. Summary of the Invention

[0004] Embodiments of this disclosure provide a cloud positioning method, apparatus, device, and storage medium for evaluating cloud positioning results to improve the visual effect of image frames after fusion processing.

[0005] In a first aspect, embodiments of this disclosure provide a cloud-based positioning method, comprising:

[0006] Acquire the image frame to be located in the video;

[0007] Based on the database, the pose of the image frame to be located is calculated;

[0008] Based on the result of the pose calculation, determine whether the image frame to be located is a frame with a calculation error;

[0009] When the image frame to be located is determined to be a frame with a malfunctioning resolution, the result of the pose calculation is discarded; and

[0010] When it is determined that the image frame to be located is not a non-error image frame, the image frame to be located is located based on the result of the pose calculation.

[0011] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation includes:

[0012] Based on the interior points of the image frame to be located obtained from the pose calculation, it is determined whether the image frame to be located is a calculated erroneous image frame.

[0013] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes:

[0014] The image frame to be located is divided into multiple grids;

[0015] Based on the number of grid points occupied by the interior points of the image frame to be located, it is determined whether the image frame to be located is a frame with a solution error.

[0016] Optionally, determining that the image frame to be located is a resolution error image frame based on the number of grid points occupied by the interior points of the image frame to be located includes:

[0017] When the number of grid points occupied by the interior points is less than a preset first threshold, the image frame to be located is determined to be a solution error image frame.

[0018] Optionally, the grid satisfies predetermined conditions, which include:

[0019] The shape is a square and / or based on pixel division.

[0020] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes:

[0021] Obtain the distribution of the inliers on the image frame to be located;

[0022] Based on the distribution of the inliers on the image frame to be located, determine the degree of concentration of the inliers on the image frame to be located; and

[0023] Based on the concentration level, it is determined whether the image frame to be located is a processing error image frame.

[0024] Optionally, determining whether the image frame to be located is a processing error image frame based on the concentration level includes:

[0025] When the ratio of the area occupied by the interior point to the area of ​​the image frame to be located is less than the second threshold, the image frame to be located is determined to be a solution error image frame.

[0026] Optionally, obtaining the distribution of the inlier points on the image frame to be located includes:

[0027] Obtain the smallest rectangle containing multiple interior points; wherein the area occupied by each interior point is the area of ​​the smallest rectangle containing multiple interior points.

[0028] Optionally, before determining whether the image frame to be located is a processing error image frame based on the concentration level, the method further includes:

[0029] The second threshold is adjusted based on the target distance between the center of the smallest rectangle containing multiple interior points and the center of the image frame to be located, wherein the target distance is positively correlated with the second threshold.

[0030] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation includes:

[0031] The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0032] Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0033] Based on the first pose and the second pose, the end-position pose difference is obtained;

[0034] Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0035] Secondly, embodiments of this disclosure provide a cloud-based positioning device, comprising:

[0036] The acquisition unit is configured to acquire the image frame to be located in the video.

[0037] The calculation unit is configured to perform pose calculation on the image frame to be located based on the database.

[0038] The determining unit is configured to determine whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation; when the image frame to be located is determined to be a frame with a calculation error, discard the result of the pose calculation; and when the image frame to be located is determined to be a frame with a non-calculation error, locate the image frame to be located based on the result of the pose calculation.

[0039] Optionally, the determining unit is configured to determine whether the image frame to be located is a miscalculated image frame based on the result of the pose calculation, including:

[0040] Based on the interior points of the image frame to be located obtained from the pose calculation, it is determined whether the image frame to be located is a frame with a calculation error.

[0041] Optionally, the determining unit is configured to determine whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation, including:

[0042] The image frame to be located is divided into multiple grids;

[0043] Based on the number of grid points occupied by the interior points of the image frame to be located, it is determined whether the image frame to be located is a frame with a solution error.

[0044] Optionally, the determining unit is configured to determine the image frame to be located as a solution error image frame based on the number of inliers occupying the grid in the image frame to be located, including:

[0045] When the number of grid points occupied by the interior points is less than a preset first threshold, the image frame to be located is determined to be a solution error image frame.

[0046] Optionally, the grid satisfies predetermined conditions, which include:

[0047] The shape is a square and / or based on pixel division.

[0048] Optionally, the determining unit is configured to determine whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation, including:

[0049] Obtain the distribution of the inliers on the image frame to be located;

[0050] Based on the distribution of the inliers on the image frame to be located, determine the degree of concentration of the inliers on the image frame to be located; and

[0051] Based on the concentration level, it is determined whether the image frame to be located is a processing error image frame.

[0052] Optionally, the determining unit is configured to determine whether the image frame to be located is a processing error image frame based on the concentration level, including:

[0053] When the ratio of the area occupied by the interior point to the area of ​​the image frame to be located is less than the second threshold, the image frame to be located is determined to be a solution error image frame.

[0054] Optionally, the determining unit is configured to acquire the distribution of the inlier points on the image frame to be located, including:

[0055] Obtain the smallest rectangle containing multiple interior points; wherein the area occupied by each interior point is the area of ​​the smallest rectangle containing multiple interior points.

[0056] Optionally, the determining unit is further configured to:

[0057] The second threshold is adjusted based on the target distance between the center of the smallest rectangle containing multiple interior points and the center of the image frame to be located, wherein the target distance is positively correlated with the second threshold.

[0058] Optionally, the determining unit is configured to determine whether the image frame to be located is a miscalculated image frame based on the result of the pose calculation, including:

[0059] The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0060] Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0061] Based on the first pose and the second pose, the end-position pose difference is obtained;

[0062] Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0063] Thirdly, embodiments of this disclosure provide a cloud positioning device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to implement the cloud positioning method as described above.

[0064] Fourthly, embodiments of this disclosure provide a computer-readable storage medium configured to store computer-executable instructions that, when executed by a processor, implement the cloud positioning method as described above.

[0065] Through one or more embodiments of this disclosure, after the cloud-based positioning pose calculation is completed, the success of the current image frame pose calculation is determined based on the number of grid points occupied by interior points, and / or based on the concentration of interior point distribution, and / or based on the difference between the pose values ​​of the calculated cloud-based positioning image frame and the previous cloud-based positioning image frame, and the difference between the pose values ​​of the corresponding two image frames on the device. When an image frame is determined to be a frame with a calculation error, the pose calculation result is discarded (i.e., it is not sent to the device for positioning); while when an image frame is determined not to be a frame with a calculation error, the image frame is positioned based on the pose calculation result. Therefore, the drawback of cloud positioning information errors introduced by directly sending cloud positioning results to the device without any evaluation in the prior art is prevented, thus improving the visual jitter caused by the fusion result on the device. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in one or more embodiments of this disclosure or in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1A A schematic flowchart illustrating the cloud positioning method provided in the first embodiment of this disclosure;

[0068] Figure 1B A schematic flowchart illustrating the cloud positioning method provided in the second embodiment of this disclosure;

[0069] Figure 2A A schematic flowchart illustrating the cloud positioning method provided in the third embodiment of this disclosure;

[0070] Figure 2B A schematic flowchart illustrating the cloud positioning method provided in the fourth embodiment of this disclosure;

[0071] Figure 3 A schematic flowchart illustrating the cloud positioning method provided in the fifth embodiment of this disclosure;

[0072] Figure 4 A schematic diagram of the structure of a cloud positioning device provided in an embodiment of this disclosure;

[0073] Figure 5A A schematic diagram of the structure of a cloud positioning device provided in yet another embodiment of this disclosure;

[0074] Figure 5B A schematic diagram of the structure of a cloud positioning device provided in another embodiment of this disclosure; and

[0075] Figure 6 This is a schematic diagram of the structure of a cloud positioning device provided in an embodiment of the present disclosure. Detailed Implementation

[0076] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this disclosure, the technical solutions in one or more embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on one or more embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0077] The execution subject of the edge-cloud converged positioning method of this application can be an edge-cloud converged positioning device. For example, the positioning method of this application can be executed by a terminal device, a cloud device, a server, or other processing device. The terminal device can be a mobile device such as a robot, an unmanned vehicle, or a drone, or a user equipment (UE), a user terminal, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc.

[0078] It should be noted that, in this application, interior points refer to feature points whose reprojection error is less than a certain threshold. In the technical solution of this application, after the cloud-based positioning pose calculation is successful, the success of the current pose calculation is determined based on the number of grid points occupied by the interior points and their degree of concentration.

[0079] In computer vision, there are four main coordinate systems: the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. The image coordinate system is established based on a two-dimensional photograph taken by a camera. It specifies the position of an object in the photograph and represents the perspective projection of the object from the camera coordinate system to the image coordinate system. The origin of this coordinate system is located at the intersection O of the camera's optical axis and the image plane, usually at the center of the photograph. The X-axis points horizontally to the right, and the Y-axis points vertically upwards. The camera coordinate system measures the object from its own perspective. Its origin is the camera's optical center. The Xc and Yc axes are parallel to the x and y axes of the image coordinate system, and the Zc axis is the camera's optical axis, perpendicular to the image plane and passing through the origin of the image coordinate system (the perpendicular intersection of the camera's optical axis and the image plane). The world coordinate system is the reference coordinate system for all coordinate systems in real space. In computer vision, it is used to describe the position of the camera and objects and does not change with the state of the camera or objects; it is always objectively present. The world coordinate system has the Earth's center of mass as its origin Ow, the Yw axis points to the geomagnetic north pole (downward), the Z axis is opposite to the direction of gravity (pointing towards the sky), and the X axis is the cross product of Y and Z (which can be determined by the right-hand rule).

[0080] Pose refers to the position and orientation of an object. Position data includes latitude, longitude, and altitude; orientation includes azimuth, pitch, and roll angles. An object's position can be represented by (x, y, z), while its orientation can be represented by (α, β, γ), which represent the angles of rotation around the three coordinate axes. Therefore, pose includes both position and orientation, representing a 6-DOF state.

[0081] There are various pose estimation methods, and the most suitable method should be selected for different conditions. Among them, the most commonly used is pose measurement based on the PnP problem. The PnP problem is how to determine the camera pose, or the pose of the point pairs in the camera coordinate system, given a 3D-2D point pair. It describes how to estimate the camera pose when the coordinates of n 3D points and their 2D projection positions are known. The description and definition of the PnP problem are relatively simple; its purpose is to solve for the motion of 3D-2D point pairs. Simply put, it's about how to estimate the camera pose (i.e., the camera's pose in coordinate system A) given the coordinates of n 3D points (relative to a specified coordinate system A) and their 2D projection positions. For example, if we know the relative coordinates of at least four points in an image in 3D space, we can estimate the camera pose relative to these points, or the pose of these 3D points in the camera coordinate system. There are many common methods for solving PNP problems, such as direct linear transformation, EPNP method, and nonlinear optimization methods. The specific pose calculation method can be selected according to the application requirements, and will not be elaborated here.

[0082] Reference Figure 1A As shown, Figure 1A A flowchart illustrating the cloud positioning method provided in the first embodiment of this disclosure is shown below. Figure 1A As shown, the process includes the following steps:

[0083] Step 101A: Obtain the image frame to be located in the video;

[0084] Step 102A: Based on the database, perform pose calculation on the image frame to be located;

[0085] Step 103A: Based on the result of the pose calculation, determine whether the image frame to be located is a frame with a calculation error;

[0086] Step 104A: When it is determined that the image frame to be located is a solution error image frame, the result of the pose calculation is discarded; and when it is determined that the image frame to be located is not a solution error image frame, the image frame to be located is located based on the result of the pose calculation.

[0087] In this embodiment, determining whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation includes:

[0088] Based on the interior points of the image frame to be located obtained from the pose calculation, it is determined whether the image frame to be located is a calculated erroneous image frame.

[0089] In this embodiment, determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes:

[0090] The image frame to be located is divided into multiple grids;

[0091] Based on the number of grid points occupied by the interior points of the image frame to be located, it is determined whether the image frame to be located is a frame with a solution error.

[0092] In this embodiment, determining that the image frame to be located is a resolution error image frame based on the number of grid points occupied by the interior points of the image frame to be located includes:

[0093] When the number of grid points occupied by the interior points is less than a preset first threshold, the image frame to be located is determined to be a solution error image frame.

[0094] In this embodiment, the grid satisfies predetermined conditions, which include:

[0095] The shape is a square and / or based on pixel division.

[0096] In this embodiment, determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes:

[0097] Obtain the distribution of the inliers on the image frame to be located;

[0098] Based on the distribution of the inliers on the image frame to be located, determine the degree of concentration of the inliers on the image frame to be located; and

[0099] Based on the concentration level, it is determined whether the image frame to be located is a processing error image frame.

[0100] In this embodiment, determining whether the image frame to be located is a processing error image frame based on the concentration level includes:

[0101] When the ratio of the area occupied by the interior point to the area of ​​the image frame to be located is less than the second threshold, the image frame to be located is determined to be a solution error image frame.

[0102] In this embodiment, obtaining the distribution of the inlier points on the image frame to be located includes:

[0103] Obtain the smallest rectangle containing multiple interior points; wherein the area occupied by each interior point is the area of ​​the smallest rectangle containing multiple interior points.

[0104] In this embodiment, before determining whether the image frame to be located is a processing error image frame based on the concentration level, the method further includes:

[0105] The second threshold is adjusted based on the target distance between the center of the smallest rectangle containing multiple interior points and the center of the image frame to be located, wherein the target distance is positively correlated with the second threshold.

[0106] In this embodiment, determining whether the image frame to be located is a miscalculated image frame based on the pose calculation result includes:

[0107] The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0108] Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0109] Based on the first pose and the second pose, the end-position pose difference is obtained;

[0110] Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0111] Reference Figure 1B As shown, Figure 1B A flowchart illustrating the cloud positioning method provided in the second embodiment of this disclosure is shown below. Figure 1B As shown, the process includes the following steps:

[0112] Step 101B: Divide the image frame into multiple grids;

[0113] Step 102B: Determine whether the image frame is a solution error image frame based on the number of grid points occupied by the interior points.

[0114] In this embodiment, determining that an image frame is a processing error image frame based on the number of grid points occupied by interior points includes:

[0115] When it is determined that the number of grid points occupied by interior points is less than a preset first threshold, the image frame is identified as a solution error image frame.

[0116] In this embodiment, the grid is square and is divided based on pixels. In a preferred embodiment of this application, one pixel can be used as the grid size.

[0117] In yet another embodiment of this application, the method further includes:

[0118] When it is determined that the number of grid cells occupied by interior points is greater than or equal to a preset first threshold, the distribution of interior points on the image frame is obtained.

[0119] Based on the distribution of inliers on the image frame, determine the degree of concentration of inliers on the image frame; and

[0120] Based on the concentration of interior points on the image frame, determine whether the current image frame is a solution error frame.

[0121] In another embodiment of this application, determining whether an image frame is a processing error image frame based on the concentration of inliers on the image frame includes: comparing the ratio of the area occupied by the inliers to the area of ​​the image frame with a preset second threshold; if the ratio of the area occupied by the inliers to the area of ​​the entire image frame is less than the second threshold, then the image frame is determined to be a processing error image frame.

[0122] In another embodiment of this application, obtaining the distribution of interior points on an image frame includes: obtaining the smallest rectangle containing all interior points; the area occupied by the interior points is the area of ​​the smallest rectangle containing all interior points.

[0123] In another embodiment of this application, the second threshold adopts an adaptive threshold method, that is, the second threshold is automatically adjusted according to the distance between the center of the smallest rectangle containing all interior points and the center of the image frame. The closer the distance between the center of the smallest rectangle containing all interior points and the center of the image frame, the smaller the second threshold.

[0124] In this embodiment, the success of the current image frame pose calculation is determined based on the number of grid points occupied by the interior points, or further based on the degree of concentration of the interior point distribution. This prevents the image frames with incorrect calculation results from being sent to the device, thus avoiding the drawbacks of directly sending cloud positioning information and introducing errors in the prior art.

[0125] In yet another embodiment of this application, the method further includes:

[0126] The pose difference in the cloud is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0127] Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0128] Based on the first pose and the second pose, the end-position pose difference is obtained;

[0129] Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0130] The difference in pose values ​​between the current cloud-based positioning image frame and the previous cloud-based positioning image frame is subtracted from the difference in pose values ​​between the corresponding two image frames on the device. The subtraction result is then compared with a third threshold. If the subtraction result is greater than the third threshold, the image frame is determined to be a frame with a calculation error.

[0131] In this embodiment, the cloud-based pose calculation result is evaluated by evaluating the local relative pose on the device. Since the device-based pose calculation result is usually accurate, the relative pose difference has some reference value. By subtracting the relative pose difference on the cloud from the relative pose difference on the device, the difference between them can be measured. The subtraction result is then compared with a third threshold, which can be set empirically in practice. If the subtraction result is greater than the third threshold, the current image frame is determined to be a erroneous frame and is not sent to the device. This further refines the image frame calculation result, improving the visual jitter caused by the fusion result on the device.

[0132] Reference Figure 2A As shown, Figure 2A A flowchart illustrating the cloud positioning method provided in the third embodiment of this disclosure is shown below. Figure 2A As shown, the process includes the following steps:

[0133] Step 201A: Obtain the distribution of inliers on the image frame;

[0134] Step 202A: Determine the degree of concentration of inliers on the image frame based on the distribution of inliers on the image frame;

[0135] Step 203A: Based on the concentration of interior points on the image frame, determine whether the image frame is a solution error image frame.

[0136] In one embodiment of this application, step 203A, which determines whether an image frame is a solution error image frame based on the concentration of interior points on the image frame, includes: when it is determined that the concentration of interior points on the image frame is less than a preset second threshold, then the image frame is determined to be a solution error image frame.

[0137] In one embodiment of this application, when it is determined that the concentration of inliers on an image frame is less than a preset second threshold, the image frame is determined to be a processing error image frame, including: comparing the proportion of the area occupied by inliers to the area of ​​the image frame with the preset second threshold, and when the proportion of the area occupied by inliers to the area of ​​the entire image frame is less than the second threshold, the image frame is determined to be a processing error image frame.

[0138] In another embodiment of this application, obtaining the distribution of interior points on an image frame includes: obtaining the smallest rectangle containing all interior points; the area occupied by the interior points is the area of ​​the smallest rectangle containing all interior points.

[0139] In another embodiment of this application, the second threshold adopts an adaptive threshold method, that is, the second threshold is automatically adjusted according to the distance between the center of the smallest rectangle containing all interior points and the center of the image frame. The closer the distance between the center of the smallest rectangle containing all interior points and the center of the image frame, the smaller the second threshold.

[0140] In this embodiment, the success of the current image frame pose calculation is determined based on the concentration of interior point distribution. Therefore, image frames with incorrect calculation results are not sent to the device, thus preventing the drawback of directly sending cloud positioning information and introducing errors as in the prior art.

[0141] In yet another embodiment of this application, the method further includes:

[0142] The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0143] Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0144] Based on the first pose and the second pose, the end-position pose difference is obtained;

[0145] Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0146] The difference in pose values ​​between the current cloud-based positioning image frame and the previous cloud-based positioning image frame is subtracted from the difference in pose values ​​between the corresponding two image frames on the device. The subtraction result is then compared with a third threshold. If the subtraction result is greater than the third threshold, the current image frame is determined to be a frame with a calculation error.

[0147] In this embodiment, the cloud-based pose calculation result is evaluated by evaluating the local relative pose on the device. Since the device-based pose calculation result is usually accurate, the relative pose difference has some reference value. By subtracting the relative pose difference on the cloud from the relative pose difference on the device, the difference between them can be measured. The subtraction result is then compared with a third threshold, which can be set empirically in practice. If the subtraction result is greater than the third threshold, the current image frame is determined to be a erroneous frame and is not sent to the device. This further refines the image frame calculation result, improving the visual jitter caused by the fusion result on the device.

[0148] Reference Figure 2B As shown, Figure 2B A flowchart illustrating the cloud positioning method provided in the fourth embodiment of this disclosure is shown below. Figure 2B As shown, the process includes the following steps:

[0149] Step 201B: Obtain the pose difference in the cloud based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0150] Step 202B: Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0151] Step 203B: Based on the first pose and the second pose, obtain the end-position pose difference;

[0152] Step 204B: Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0153] In this embodiment, the cloud-based pose calculation result is evaluated by evaluating the local relative pose on the device. Since the device-based pose calculation result is usually accurate, the relative pose difference has some reference value. By subtracting the relative pose difference on the cloud from the relative pose difference on the device, the difference between them can be measured. The subtraction result is then compared with a third threshold, which can be set empirically in practice. If the subtraction result is greater than the third threshold, the current image frame is determined to be a erroneous frame and is not sent to the device. This further refines the image frame calculation result, improving the visual jitter caused by the fusion result on the device.

[0154] Reference Figure 3 As shown, Figure 3 A flowchart illustrating the cloud positioning method provided in the fifth embodiment of this disclosure is shown below. Figure 3 As shown, the process includes the following steps:

[0155] Step 301: Divide the current image frame into a certain number of grids;

[0156] Step 302: Determine whether the number of grid cells occupied by the inner points is less than the preset first threshold. If it is determined that the number of grid cells occupied by the inner points is less than the preset first threshold, determine that the current image frame is in error and proceed to step 306; otherwise, proceed to step 303.

[0157] In this embodiment, the current image frame is divided into a certain number of grids, and each grid is square. The size of the grid cells can be determined based on pixels; in a preferred embodiment of this application, one pixel can be used as the grid size.

[0158] Taking an image frame with a pixel value of 720×1280 as an example, there are a total of 720×1280 grids. In this embodiment, the grid size can range from 1 to 5 pixels; the larger the grid, the faster the processing speed. It should be noted that using one pixel as the grid size is more reliable and more accurate in identifying erroneous frames.

[0159] The first threshold is determined by the number of points required to calculate the accurate pose, and can be set differently according to the actual application needs. The selectable value range for this threshold is 6-7. For a single grid cell, all points within that cell are considered to contribute 1 to the pose calculation, because the points within a single cell are too close together; the contribution of all interior points within a cell to the pose calculation result can be considered equivalent to the contribution of a single interior point. Typically, a single image frame can extract several thousand 2D points, such as 4000. By matching these with 3D points, 200-300 interior points are obtained, and then these 200-300 interior points are used to divide the image into grids. When it is determined that the number of grids occupied by these interior points is less than the threshold, i.e., 6-7, since the contribution of all interior points in a pixel to the pose calculation can be considered unique, if these interior points are concentrated in these 6-7 pixels, these interior points are too concentrated and do not contribute much to the overall pose calculation result. Therefore, it is confirmed that the current image frame calculation is wrong and will not be sent to the device for registration.

[0160] Of course, there may be more than a dozen matching interior points, and the judgment method is the same as above, so this application will not repeat it.

[0161] However, there exists a situation where the number of grid cells occupied by an interior point exceeds a preset first threshold, meaning the number of grid cells occupied by the interior point meets the set requirements, but the positions of these occupied grid cells on the image frame are too concentrated. In this case, the result of pose calculation will not be particularly ideal. Therefore, this application introduces the following steps:

[0162] Step 303: Obtain the distribution of inliers on the image frame, and determine the degree of concentration of inliers on the image frame based on the distribution of inliers on the image frame.

[0163] Step 304: Based on the concentration of the inliers on the image frame, determine whether the current image frame is erroneous. Compare the ratio of the area occupied by the inliers to the area of ​​the entire image frame with a preset second threshold. If the ratio of the area occupied by the inliers to the area of ​​the entire image frame is less than the second threshold, then determine that the current image frame is erroneous and proceed to step 306; otherwise, proceed to step 305.

[0164] In this embodiment, the success of the pose calculation for the current image frame is determined based on the degree of concentration of the interior points on the image frame. The distribution of interior points mainly depends on 2D points, and the degree of concentration of the interior points on the image frame is measured by the area occupied by the interior points.

[0165] In this embodiment, the area occupied by the interior points is described by a rectangle. Specifically, the rectangle is determined as follows: traverse all interior points, and the four outermost interior points determine four lines. These four lines then determine the smallest rectangle containing all interior points. It can be seen that the area occupied by the interior points, which is the area of ​​the smallest rectangle containing all interior points, mainly depends on the leftmost, rightmost, topmost, and bottommost interior points among all interior points in the image frame. These interior points determine the boundary of the rectangle. The larger the rectangle area, the more dispersed the distribution; the smaller the rectangle area, the more concentrated the distribution.

[0166] The proportion of the area occupied by the inlier to the total area of ​​the image frame is compared with a preset second threshold. If the proportion of the area occupied by the inlier to the total area of ​​the image frame is less than the second threshold, the pose calculation of the current image frame is determined to have failed, and the current image frame is determined to have an error.

[0167] In this embodiment, the second threshold is an adaptive threshold, meaning it is automatically adjusted based on the distance between the center of the smallest rectangle containing all inliers (hereinafter referred to as the inlier rectangle) and the center of the image frame. The distance to the image frame center primarily depends on the distance from the center of the inlier rectangle to the image frame origin. When the center of the inlier rectangle is closer to the image frame origin, the second threshold is smaller; conversely, the farther the center of the inlier rectangle is from the image frame origin, the larger the second threshold. This is because, for an image frame, inliers at the edges provide a poor overall description of the image frame and contribute little to the pose calculation of the current frame. For example, these inliers might be located at the top left, bottom left, top right, or bottom right corners of the image frame. However, inliers located closer to the image frame origin provide a better description of the image frame and contribute more to the pose calculation results. For an image frame, when all interior points are near the origin of the image frame, when three-dimensional points are projected onto two-dimensional points, a connection is established between them, and there will not be much deviation in pose calculation; when all interior points are on the edge, if the area where they are concentrated is larger, it will be more helpful for pose calculation.

[0168] In one application example of this application, when the center of the inner point rectangle is at the origin of the image frame (i.e., coincides with the origin), the second threshold is 1 / 40; when the center of the inner point rectangle is at the edge of the image frame, the second threshold is 1 / 20. That is, for points closer to the origin of the image frame, the threshold requirement is lower. For points where the center of the inner point rectangle is any point between the origin and the edge of the image frame, the second threshold can be obtained by the ratio of the pixel value of that arbitrary point to the pixel value of the image frame edge. For example, if the pixel value of the center of the inner point rectangle is 240, and the pixel value of the image frame edge is 360, then the corresponding second threshold is 240 divided by 360 multiplied by 1 / 20. Of course, the second threshold can be set in other ways based on experience. Any numerical range that can distinguish the concentration of the inner point layout is within the protection scope of this application, and this application does not limit it in this regard.

[0169] It is understandable that the center of an image frame can be determined in various ways. For example, for an image frame with a pixel value of 720×1280, the center of the image frame can be the pixel value divided by 2, that is, the point with a pixel value of 360×640.

[0170] Step 305: Subtract the difference in pose values ​​between the current cloud positioning frame and the previous cloud positioning frame from the difference in pose values ​​between the corresponding two frames on the device to obtain the subtraction result. Compare the subtraction result with the third threshold. If the subtraction result is greater than the third threshold, the image frame is determined to be a miscalculated image frame, and step 306 is executed.

[0171] In this embodiment, the local relative pose on the device is used to evaluate the pose calculation results on the cloud.

[0172] Step 306: The current image frame is not sent to the device as a registration frame, and the process ends.

[0173] In this embodiment, after the cloud-based positioning pose calculation is successful, the success of the current image frame pose calculation is determined based on the number of grids occupied by the interior points and the degree of distribution concentration. Therefore, image frames with incorrect calculation results are not sent to the device, which prevents the drawback of directly sending cloud positioning information and introducing errors in the prior art. This improves the visual jitter caused by the fusion results on the device.

[0174] In this embodiment, the cloud-based pose calculation result is evaluated by evaluating the local relative pose on the device. Since the device-based pose calculation result is usually accurate, the relative pose difference has some reference value. By subtracting the relative pose difference on the cloud from the relative pose difference on the device, the difference between them can be measured. The subtraction result is then compared with a third threshold, which can be set empirically in practice. If the subtraction result is greater than the third threshold, the current image frame is determined to be a erroneous frame and is not sent to the device. This further refines the image frame calculation result, improving the visual jitter caused by the fusion result on the device.

[0175] Figure 4 This is a schematic diagram of the structure of a cloud positioning device provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, the device includes:

[0176] The acquisition unit 401 is configured to acquire the image frame to be located in the video;

[0177] The calculation unit 402 is configured to perform pose calculation on the image frame to be located based on the database.

[0178] The determining unit 403 is configured to determine whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation; when the image frame to be located is determined to be a frame with a calculation error, discard the result of the pose calculation; and when the image frame to be located is determined to be a frame with a calculation error, locate the image frame to be located based on the result of the pose calculation.

[0179] In this embodiment, the determining unit 403 is configured to determine whether the image frame to be located is a miscalculated image frame based on the result of the pose calculation, including:

[0180] Based on the interior points of the image frame to be located obtained from the pose calculation, it is determined whether the image frame to be located is a frame with a calculation error.

[0181] In this embodiment, the determining unit 403 is configured to determine whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained by the pose calculation.

[0182] The image frame to be located is divided into multiple grids;

[0183] Based on the number of grid points occupied by the interior points of the image frame to be located, it is determined whether the image frame to be located is a frame with a solution error.

[0184] Optionally, the determining unit 403 is configured to determine the image frame to be located as a solution error image frame based on the number of inliers occupying the grid in the image frame to be located, including:

[0185] When the number of grid points occupied by the interior points is less than a preset first threshold, the image frame to be located is determined to be a solution error image frame.

[0186] Optionally, the grid satisfies predetermined conditions, which include:

[0187] The shape is a square and / or based on pixel division.

[0188] Optionally, the determining unit 403 is configured to determine whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation, including:

[0189] Obtain the distribution of the inliers on the image frame to be located;

[0190] Based on the distribution of the inliers on the image frame to be located, determine the degree of concentration of the inliers on the image frame to be located; and

[0191] Based on the concentration level, it is determined whether the image frame to be located is a processing error image frame.

[0192] Optionally, the determining unit 403 is configured to determine whether the image frame to be located is a processing error image frame based on the concentration level, including:

[0193] When the ratio of the area occupied by the interior point to the area of ​​the image frame to be located is less than the second threshold, the image frame to be located is determined to be a solution error image frame.

[0194] Optionally, the determining unit 403 is configured to acquire the distribution of the inlier points on the image frame to be located, including:

[0195] Obtain the smallest rectangle containing multiple interior points; wherein the area occupied by each interior point is the area of ​​the smallest rectangle containing multiple interior points.

[0196] Optionally, the determining unit 403 is further configured to:

[0197] The second threshold is adjusted based on the target distance between the center of the smallest rectangle containing multiple interior points and the center of the image frame to be located, wherein the target distance is positively correlated with the second threshold.

[0198] Optionally, the determining unit 403 is configured to determine whether the image frame to be located is a miscalculated image frame based on the result of the pose calculation, including:

[0199] The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0200] Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0201] Based on the first pose and the second pose, the end-position pose difference is obtained;

[0202] Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0203] The difference in pose values ​​between the current cloud-based image frame and the historically located image frames in the cloud is subtracted from the difference in pose values ​​between the corresponding two image frames on the device. The subtraction result is then compared with a third threshold. If the subtraction result is greater than the third threshold, the current image frame is determined to be a miscalculated image frame.

[0204] In this embodiment, after the cloud-based positioning pose calculation is successful, the success of the current image frame pose calculation is determined based on the number of grids occupied by the interior points and the degree of distribution concentration. Therefore, image frames with incorrect calculation results are not sent to the device, which prevents the drawback of directly sending cloud positioning information and introducing errors in the prior art. This improves the visual jitter caused by the fusion results on the device.

[0205] In this embodiment, the cloud-based pose calculation result is evaluated by evaluating the local relative pose on the device. Since the device-based pose calculation result is usually accurate, the relative pose difference has some reference value. By subtracting the relative pose difference on the cloud from the relative pose difference on the device, the difference between them can be measured. The subtraction result is then compared with a third threshold, which can be set empirically in practice. If the subtraction result is greater than the third threshold, the current image frame is determined to be a erroneous frame and is not sent to the device. This further refines the image frame calculation result, improving the visual jitter caused by the fusion result on the device.

[0206] Figure 5A This is a schematic diagram of the structure of a cloud positioning device provided in another embodiment of the present disclosure, as shown below. Figure 5A As shown, the device includes:

[0207] The partitioning unit 501 is configured to divide an image frame into a certain number of grids;

[0208] The first determining unit 502 is configured to determine whether the image frame is a solution error image frame based on the number of grids occupied by interior points.

[0209] Optionally, the first determining unit 502 is configured to determine the image frame as a computationally erroneous image frame when it is determined that the number of grid points occupied by interior points is less than a preset first threshold.

[0210] Optionally, the partitioning unit 501 is configured to divide an image frame into a certain number of grids, the grids being rectangular and based on pixel partitioning.

[0211] Optionally, the device may also include:

[0212] The second acquisition unit 505 is configured to acquire the distribution of the inner points on the image frame when it is determined that the number of grids occupied by the inner points is greater than or equal to a preset first threshold.

[0213] The third determining unit 504 is configured to determine the concentration of inlier points on an image frame based on the distribution of inlier points on the image frame; and

[0214] The second judgment unit 503 is configured to determine whether an image frame is a solved erroneous image frame based on the degree of concentration of interior points on the image frame.

[0215] Optionally, the second judgment unit 503 determines whether an image frame is a solution error image frame based on the concentration of inliers on the image frame, including: comparing the ratio of the area occupied by the inliers to the area of ​​the image frame with a preset second threshold; if the ratio of the area occupied by the inliers to the area of ​​the entire image frame is less than the second threshold, then the image frame is determined to be a solution error image frame.

[0216] Optionally, the second acquisition unit 505 acquires the distribution of interior points on the image frame, including: acquiring the smallest rectangle containing all interior points; the area occupied by the interior points is the area of ​​the smallest rectangle containing all interior points.

[0217] Optionally, the second judgment unit compares the ratio of the area occupied by the inner points to the area of ​​the image frame with a preset second threshold. The second threshold is automatically adjusted based on the distance between the center of the smallest rectangle containing all inner points and the center of the image frame. The closer the distance between the center of the smallest rectangle containing all inner points and the center of the image frame, the smaller the second threshold.

[0218] Optionally, the device further includes:

[0219] The relative pose comparison unit 506 is configured to subtract the pose value difference between the current cloud-based positioning image frame and the previous cloud-based positioning image frame from the pose value difference between the corresponding two image frames on the device obtained, and then compare the subtraction result with a third threshold.

[0220] The second judgment unit 503 is also configured to determine that the current image frame is a solution error image frame when the subtraction result is greater than the third threshold.

[0221] In this embodiment, after the cloud-based positioning pose calculation is successful, the success of the current image frame pose calculation is determined based on the number of grids occupied by the interior points and the degree of distribution concentration. Therefore, image frames with incorrect calculation results are not sent to the device, which prevents the drawback of directly sending cloud positioning information and introducing errors in the prior art. This improves the visual jitter caused by the fusion results on the device.

[0222] In this embodiment, the cloud-based pose calculation result is evaluated by evaluating the local relative pose on the device. Since the device-based pose calculation result is usually accurate, the relative pose difference has some reference value. By subtracting the relative pose difference on the cloud from the relative pose difference on the device, the difference between them can be measured. The subtraction result is then compared with a third threshold, which can be set empirically in practice. If the subtraction result is greater than the third threshold, the current image frame is determined to be a erroneous frame and is not sent to the device. This further refines the image frame calculation result, improving the visual jitter caused by the fusion result on the device.

[0223] Figure 5B A schematic diagram of the structure of a cloud positioning device provided in another embodiment of this disclosure is shown below. Figure 5B As shown, the device includes:

[0224] The first acquisition unit 508 is configured to acquire the distribution of interior points on the image;

[0225] The second determining unit 510 is configured to determine the degree of concentration of inlier points on the image based on the distribution of inlier points on the image.

[0226] The first judgment unit 509 is configured to determine whether an image is a solution error image based on the concentration of interior points on the image.

[0227] Optionally, the first judgment unit 509 is configured to determine that the image is a solution error image when it is determined that the concentration of interior points on the image is less than a preset second threshold.

[0228] Optionally, the first judgment unit 509 is configured to determine that the current image is erroneous when it is determined that the distribution of inliers on the image is less than a preset second threshold, including: comparing the proportion of the area occupied by inliers to the area of ​​the whole image with the preset second threshold; if the proportion of the area occupied by inliers to the area of ​​the whole image is less than the second threshold, then the current image is determined to be an image with erroneous resolution.

[0229] Optionally, the first acquisition unit 508 acquires the distribution of interior points on the image, including: acquiring the smallest rectangle containing all interior points; the area occupied by the interior points is the area of ​​the smallest rectangle containing all interior points.

[0230] Optionally, the first judgment unit 509 compares the proportion of the area occupied by the inner points to the area of ​​the entire image with a preset second threshold. The second threshold is automatically adjusted based on the distance between the center of the smallest rectangle containing all inner points and the center of the image. The closer the distance between the center of the smallest rectangle containing all inner points and the center of the image, the smaller the second threshold.

[0231] Optionally, the device further includes:

[0232] The second relative pose comparison unit 511 is configured to subtract the pose value difference between the calculated current cloud-based positioning image and the historical cloud-based positioning image from the pose value difference between the corresponding two frames of images on the device, obtain the subtraction result, and compare the subtraction result with the third threshold.

[0233] The first judgment unit 509 is also configured to determine that the current image is a solution error image when the comparison result is greater than the third threshold.

[0234] The cloud positioning device in the embodiments of this disclosure can implement the various processes of the aforementioned cloud positioning method embodiments and achieve the same effects and functions, which will not be repeated here.

[0235] One embodiment of this disclosure also provides a cloud-based positioning device. Figure 6 A schematic diagram of the structure of a cloud positioning device provided in an embodiment of this disclosure is shown below. Figure 6 As shown, cloud-based positioning devices can vary significantly due to differences in configuration or performance. They may include one or more processors 601 and memory 602, with memory 602 storing one or more applications or data. Memory 602 can be temporary or persistent storage. Applications stored in memory 602 may include one or more modules (not shown), each module including a series of computer-executable instructions for the positioning device. Furthermore, processor 601 may be configured to communicate with memory 602, executing the series of computer-executable instructions in memory 602 on the positioning device. The positioning device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input or output interfaces 605, one or more keyboards 606, etc. It is understood that... Figure 6 This is just one possible example; in real-world applications, positioning devices can be configured arbitrarily as needed.

[0236] In one specific embodiment, the cloud-based positioning device includes a processor; and a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the following processes:

[0237] Acquire the image frame to be located in the video;

[0238] Based on the database, the pose of the image frame to be located is calculated;

[0239] Based on the result of the pose calculation, determine whether the image frame to be located is a frame with a calculation error;

[0240] When the image frame to be located is determined to be a frame with a malfunctioning resolution, the result of the pose calculation is discarded; and

[0241] When it is determined that the image frame to be located is not a non-error image frame, the image frame to be located is located based on the result of the pose calculation.

[0242] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation includes:

[0243] Based on the interior points of the image frame to be located obtained from the pose calculation, it is determined whether the image frame to be located is a calculated erroneous image frame.

[0244] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes:

[0245] The image frame to be located is divided into multiple grids;

[0246] Based on the number of grid points occupied by the interior points of the image frame to be located, it is determined whether the image frame to be located is a frame with a solution error.

[0247] Optionally, determining that the image frame to be located is a resolution error image frame based on the number of grid points occupied by the interior points of the image frame to be located includes:

[0248] When the number of grid points occupied by the interior points is less than a preset first threshold, the image frame to be located is determined to be a solution error image frame.

[0249] Optionally, the grid satisfies predetermined conditions, which include:

[0250] The shape is a square and / or based on pixel division.

[0251] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes:

[0252] Obtain the distribution of the inliers on the image frame to be located;

[0253] Based on the distribution of the inliers on the image frame to be located, determine the degree of concentration of the inliers on the image frame to be located; and

[0254] Based on the concentration level, it is determined whether the image frame to be located is a processing error image frame.

[0255] Optionally, determining whether the image frame to be located is a processing error image frame based on the concentration level includes:

[0256] When the ratio of the area occupied by the interior point to the area of ​​the image frame to be located is less than the second threshold, the image frame to be located is determined to be a solution error image frame.

[0257] Optionally, obtaining the distribution of the inlier points on the image frame to be located includes:

[0258] Obtain the smallest rectangle containing multiple interior points; wherein the area occupied by each interior point is the area of ​​the smallest rectangle containing multiple interior points.

[0259] Optionally, before determining whether the image frame to be located is a processing error image frame based on the concentration level, the method further includes:

[0260] The second threshold is adjusted based on the target distance between the center of the smallest rectangle containing multiple interior points and the center of the image frame to be located, wherein the target distance is positively correlated with the second threshold.

[0261] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation includes:

[0262] The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0263] Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0264] Based on the first pose and the second pose, the end-position pose difference is obtained;

[0265] Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0266] In this embodiment, after the cloud-based positioning pose calculation is successful, the success of the current image frame pose calculation is determined based on the number of grids occupied by the interior points and the degree of distribution concentration. Therefore, image frames with incorrect calculation results are not sent to the device, which prevents the drawback of directly sending cloud positioning information and introducing errors in the prior art. This improves the visual jitter caused by the fusion results on the device.

[0267] In this embodiment, the cloud-based pose calculation result is evaluated by evaluating the local relative pose on the device. Since the pose calculation result on the device is usually quite accurate, the relative pose difference has some reference value. By subtracting the relative pose difference on the cloud from the relative pose difference on the device, the difference between them can be measured. The subtraction result is then compared with a third threshold, which can be set empirically in practice. If the subtraction result is greater than the third threshold, the current image is determined to be a erroneous image and is not sent to the device. This further refines the image calculation result, improving the visual jitter caused by the fusion result on the device.

[0268] The cloud positioning device in this embodiment can implement all the processes of the aforementioned cloud positioning method embodiments and achieve the same effects and functions, which will not be repeated here.

[0269] One embodiment of this disclosure also provides a computer-readable storage medium configured to store computer-executable instructions that, when executed by a processor, implement the following process:

[0270] Acquire the image frame to be located in the video;

[0271] Based on the database, the pose of the image frame to be located is calculated;

[0272] Based on the result of the pose calculation, determine whether the image frame to be located is a frame with a calculation error;

[0273] When the image frame to be located is determined to be a frame with a malfunctioning resolution, the result of the pose calculation is discarded; and

[0274] When it is determined that the image frame to be located is not a non-error image frame, the image frame to be located is located based on the result of the pose calculation.

[0275] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation includes:

[0276] Based on the interior points of the image frame to be located obtained from the pose calculation, it is determined whether the image frame to be located is a calculated erroneous image frame.

[0277] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes:

[0278] The image frame to be located is divided into multiple grids;

[0279] Based on the number of grid points occupied by the interior points of the image frame to be located, it is determined whether the image frame to be located is a frame with a solution error.

[0280] Optionally, determining that the image frame to be located is a resolution error image frame based on the number of grid points occupied by the interior points of the image frame to be located includes:

[0281] When the number of grid points occupied by the interior points is less than a preset first threshold, the image frame to be located is determined to be a solution error image frame.

[0282] Optionally, the grid satisfies predetermined conditions, which include:

[0283] The shape is a square and / or based on pixel division.

[0284] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes:

[0285] Obtain the distribution of the inliers on the image frame to be located;

[0286] Based on the distribution of the inliers on the image frame to be located, determine the degree of concentration of the inliers on the image frame to be located; and

[0287] Based on the concentration level, it is determined whether the image frame to be located is a processing error image frame.

[0288] Optionally, determining whether the image frame to be located is a processing error image frame based on the concentration level includes:

[0289] When the ratio of the area occupied by the interior point to the area of ​​the image frame to be located is less than the second threshold, the image frame to be located is determined to be a solution error image frame.

[0290] Optionally, obtaining the distribution of the inlier points on the image frame to be located includes:

[0291] Obtain the smallest rectangle containing multiple interior points; wherein the area occupied by each interior point is the area of ​​the smallest rectangle containing multiple interior points.

[0292] Optionally, before determining whether the image frame to be located is a processing error image frame based on the concentration level, the method further includes:

[0293] The second threshold is adjusted based on the target distance between the center of the smallest rectangle containing multiple interior points and the center of the image frame to be located, wherein the target distance is positively correlated with the second threshold.

[0294] Optionally, determining whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation includes:

[0295] The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud.

[0296] Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud.

[0297] Based on the first pose and the second pose, the end-position pose difference is obtained;

[0298] Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

[0299] In this embodiment, after the cloud-based positioning pose calculation is successful, the success of the current image frame pose calculation is determined based on the number of grids occupied by the interior points and the degree of distribution concentration. Therefore, image frames with incorrect calculation results are not sent to the device, which prevents the drawback of directly sending cloud positioning information and introducing errors in the prior art. This improves the visual jitter caused by the fusion results on the device.

[0300] In this embodiment, the cloud-based pose calculation result is evaluated by evaluating the local relative pose on the device. Since the pose calculation result on the device is usually quite accurate, the relative pose difference has some reference value. By subtracting the relative pose difference on the cloud from the relative pose difference on the device, the difference between them can be measured. The subtraction result is then compared with a third threshold, which can be set empirically in practice. If the subtraction result is greater than the third threshold, the current image is determined to be a erroneous image and is not sent to the device. This further refines the image calculation result, improving the visual jitter caused by the fusion result on the device.

[0301] The computer-readable storage medium in this embodiment can implement the various processes of the aforementioned cloud positioning method embodiment and achieve the same effects and functions, which will not be repeated here.

[0302] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0303] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0304] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0305] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0306] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims

1. A cloud-based positioning method, characterized in that, include: Acquire the image frame to be located in the video; Based on the database, the pose of the image frame to be located is calculated; Based on the result of the pose calculation, determine whether the image frame to be located is a frame with a calculation error; When the image frame to be located is determined to be a frame with a solution error, the result of the pose calculation is discarded. as well as When it is determined that the image frame to be located is not a non-error image frame, the image frame to be located is located based on the result of the pose calculation. The step of determining whether the image frame to be located is a miscalculated image frame based on the pose calculation result includes: The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud. Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud. Based on the first pose and the second pose, the end-position pose difference is obtained; Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

2. The method according to claim 1, characterized in that, The step of determining whether the image frame to be located is a frame with a calculation error based on the result of the pose calculation includes: Based on the interior points of the image frame to be located obtained from the pose calculation, it is determined whether the image frame to be located is a calculated erroneous image frame.

3. The method according to claim 2, characterized in that, The step of determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes: The image frame to be located is divided into multiple grids; Based on the number of grid points occupied by the interior points of the image frame to be located, it is determined whether the image frame to be located is a frame with a solution error.

4. The method according to claim 3, characterized in that, The step of determining the image frame to be located as a resolution error image frame based on the number of inliers occupying the grid in the image frame to be located includes: When the number of grid points occupied by the interior points is less than a preset first threshold, the image frame to be located is determined to be a solution error image frame.

5. The method according to claim 3, characterized in that, The grid satisfies predetermined conditions, which include: The shape is a square and / or based on pixel division.

6. The method according to claim 2, characterized in that, The step of determining whether the image frame to be located is a frame with a calculation error based on the interior points of the image frame to be located obtained from the pose calculation includes: Obtain the distribution of the inliers on the image frame to be located; Based on the distribution of the inliers on the image frame to be located, determine the degree of concentration of the inliers on the image frame to be located; and Based on the concentration level, it is determined whether the image frame to be located is a processing error image frame.

7. The method according to claim 6, characterized in that, The step of determining whether the image frame to be located is a processing error image frame based on the concentration level includes: When the ratio of the area occupied by the interior point to the area of ​​the image frame to be located is less than the second threshold, the image frame to be located is determined to be a solution error image frame.

8. The method according to claim 6 or 7, characterized in that, The step of obtaining the distribution of the inliers on the image frame to be located includes: Obtain the smallest rectangle containing multiple interior points; wherein the area occupied by each interior point is the area of ​​the smallest rectangle containing multiple interior points.

9. The method according to claim 7, characterized in that, Before determining whether the image frame to be located is a computationally erroneous image frame based on the concentration level, the method further includes: The second threshold is adjusted based on the target distance between the center of the smallest rectangle containing multiple interior points and the center of the image frame to be located, wherein the target distance is positively correlated with the second threshold.

10. A cloud-based positioning device, characterized in that, include: The acquisition unit is configured to acquire the image frame to be located in the video. The calculation unit is configured to perform pose calculation on the image frame to be located based on the database. The determining unit is configured to determine whether the image frame to be located is a miscalculated image frame based on the result of the pose calculation; When the image frame to be located is determined to be a frame with a solution error, the result of the pose calculation is discarded. And when it is determined that the image frame to be located is a non-error image frame, the image frame to be located is located based on the result of the pose calculation; The determining unit is configured to determine whether the image frame to be located is a miscalculated image frame based on the result of the pose calculation, including: The cloud pose difference is obtained based on the pose of the image frame to be located and the historically located image frames in the cloud. Obtain the first pose of the on-device image frame corresponding to the image frame to be located, and obtain the second pose of the on-device image frame corresponding to the historically located image frame in the cloud. Based on the first pose and the second pose, the end-position pose difference is obtained; Subtract the cloud-based pose difference from the terminal-based pose difference. If the subtraction result is greater than the third threshold, then the image frame to be located is determined to be a frame with a solution error.

11. A cloud-based positioning device, characterized in that, include: processor; as well as, The memory is configured to store computer-executable instructions that, when executed, cause the processor to implement the cloud positioning method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store computer-executable instructions that, when executed by a processor, implement the cloud positioning method as described in any one of claims 1-9.

Images