Positioning method, apparatus, device and medium of mobile device

Abstract

The present application relates to the technical field of vehicle positioning, and discloses a positioning method and device for a mobile device, equipment and a medium. The present application obtains a target image, motion information of a mobile device, size information of a vehicle cabin, identifies structural features of a vehicle cabin entrance, and extracts actual pixel coordinates of feature points in the structural features. A vehicle cabin entrance reference coordinate system is constructed with a reference point of the vehicle cabin entrance as an origin, and vehicle cabin coordinates of the plurality of feature points in the vehicle cabin entrance reference coordinate system are determined according to the size information. A second conversion relationship is determined through a first conversion relationship, the vehicle cabin coordinates and the actual pixel coordinates of the feature points, and an internal parameter of a camera. The second conversion relationship is used to determine a relative position and a heading angle of the vehicle cabin entrance reference coordinate system relative to a mobile device coordinate system, and a visual observation is constructed. Finally, the target pose information is obtained by fusing the visual observation and the motion information. The present application directly utilizes the structural features of the vehicle cabin entrance for positioning without the aid of auxiliary facilities.

Description

Technical Field

[0001] This invention relates to the field of vehicle positioning technology, and more specifically to positioning methods, devices, equipment, and media for mobile devices. Background Technology

[0002] Mobile devices can automatically locate vehicles waiting to be loaded or unloaded in the platform area. The traditional positioning method is to park the vehicles in a designated area, which is marked with QR codes, reflectors, reflective posts, etc., and mobile devices usually need to locate them based on these marks.

[0003] Deploying auxiliary facilities in the platform area has high maintenance costs, and requires vehicles to be loaded or unloaded to park precisely in designated areas. Once the parking position changes, the mobile equipment cannot be accurately located, resulting in poor adaptability. Summary of the Invention

[0004] This invention provides a positioning method, apparatus, electronic device, storage medium, and program product for mobile devices to solve the problem of poor positioning accuracy and adaptability of mobile devices in related technologies.

[0005] In a first aspect, the present invention provides a method for locating a mobile device, the method comprising: In the current cycle, the target image captured by the camera, the motion information of the mobile device, and the size information of the carriage are acquired, wherein the camera is mounted on the mobile device; Identify the structural features of the carriage entrance in the target image, where the structural features include multiple feature points; Extract the actual pixel coordinates of multiple feature points from the target image; Construct a reference coordinate system for the car entrance with the reference point of the car entrance as the origin, and determine the car coordinates of multiple feature points in the reference coordinate system based on the size information. The reference point of the car entrance is the center point of the edge of the car entrance floor. Based on the first transformation relationship between the preset camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of multiple feature points, and the camera's intrinsic parameters, the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system is determined. Based on the second transformation relationship, determine the position of the car entrance reference point in the mobile device coordinate system, and determine the heading angle of the car entrance reference coordinate system relative to the mobile device coordinate system based on the second transformation relationship. Based on the location and heading angle of the reference point at the carriage entrance, construct the visual observations for the current period; The visual observations and motion information of the current cycle are fused to obtain the target pose information, which is used to characterize the relative pose of the car entrance reference coordinate system with respect to the mobile device coordinate system.

[0006] The positioning method for mobile devices provided in this invention acquires target images captured by a camera mounted on the mobile device, motion information of the mobile device, and size information of the carriage. It then identifies structural features of the carriage entrance and constructs a reference coordinate system for the carriage entrance based on the carriage size information. The method determines the carriage coordinates of the structural features within this reference coordinate system. Based on the carriage coordinates of the structural features, actual pixel coordinates, camera intrinsic parameters, and a first transformation relationship between the camera coordinate system and the mobile device coordinate system, a second transformation relationship is determined between the reference coordinate system for the carriage entrance and the mobile device coordinate system. Furthermore, the method determines the position of the carriage entrance reference point in the mobile device coordinate system and the heading angle of the reference coordinate system relative to the mobile device coordinate system based on the second transformation relationship. Finally, it fuses the visual observations consisting of the position and heading angle with the motion information of the mobile device to obtain target pose information.

[0007] Under the above scheme, by constructing a reference coordinate system for the carriage entrance with the reference point of the carriage entrance as the origin, and using the prior size information of the carriage, the carriage coordinates of the feature points at the carriage entrance in the reference coordinate system are determined. In subsequent processes, a second transformation relationship can be determined based on the carriage coordinates and actual pixel coordinates of the feature points. This allows for positioning directly using the structural features of the carriage entrance itself without the need for external markers, avoiding the high deployment cost and periodic maintenance requirements of reflectors in traditional schemes. Furthermore, even if the vehicle to be loaded or unloaded is not precisely parked in the preset position, the mobile device can still locate itself based on the actual position and orientation of the carriage entrance without the need for other auxiliary facilities, greatly reducing deployment costs and improving adaptability in platform loading and unloading scenarios.

[0008] In one optional implementation, visual observations and motion information of the current cycle are fused to obtain target pose information, wherein the target pose information is used to characterize the relative pose of the carriage entrance reference coordinate system with respect to the mobile device coordinate system, including: Based on motion information and target pose information from the previous cycle, predict the initial relative pose information for the current cycle. Obtain the quality score of the target image; Based on the quality score, the preset visual observation noise covariance is adjusted to obtain the visual observation noise covariance for the current period. The lower the quality score, the larger the visual observation noise covariance for the current period. Based on the visual observations of the current period, the visual observation noise covariance of the current period, and the initial relative pose information of the current period, the target pose information of the current period is determined.

[0009] In this way, the proposed solution obtains the quality score of the target image and adjusts the visual observation noise covariance of the current cycle based on the quality score. The visual observation noise covariance is used to adjust the initial relative pose information, which is determined based on the target pose information of the previous cycle and the motion information of the current cycle. Therefore, when the target image quality score is high, the localization result can rely more on visual information; when the target image quality score is low, the system reduces the weight of visual observations, making the localization result rely more on motion information and avoiding inaccurate localization due to erroneous information.

[0010] In one optional implementation, based on a preset first transformation relationship between the camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of multiple feature points, and the camera's intrinsic parameters, a second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system is determined, including: Based on the carriage coordinates and actual pixel coordinates of multiple feature points, as well as the camera's intrinsic parameters, a third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system is determined. Based on the first and third transformation relations, determine the second transformation relation.

[0011] In this way, by identifying feature points, and based on the carriage coordinates and actual pixel coordinates of the feature points, as well as the camera's intrinsic parameters, the third transformation relationship can be determined. The first transformation relationship is a known and definite transformation relationship. Therefore, based on the first and third transformation relationships, the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system can be determined. The visual recognition results are transformed from the image plane and camera coordinate system to the mobile device coordinate system, which is used to determine the relative positional relationship between the carriage and the mobile device. This can guide the alignment, motion control, and path planning of the mobile device without any markings.

[0012] In one optional implementation, a third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system is determined based on the carriage coordinates and actual pixel coordinates of multiple feature points, as well as the camera's intrinsic parameters. This includes: Obtain the initial transformation relationship between the car entrance reference coordinate system and the camera coordinate system, and use the initial transformation relationship as the initial value of the candidate transformation relationship; Based on the current candidate transformation relationship, the camera's intrinsic parameters, and the carriage coordinates of multiple feature points, the predicted pixel coordinates of multiple feature points are determined respectively. Based on the predicted and actual pixel coordinates of multiple feature points, determine the reprojection error corresponding to each feature point. Feature points with reprojection errors greater than a preset error threshold are removed to obtain multiple remaining feature points; Based on the total reprojection error corresponding to multiple remaining feature points, the candidate transformation relationship is optimized to obtain the third transformation relationship.

[0013] In one optional implementation, the candidate transformation relationship is optimized based on the total reprojection error corresponding to multiple remaining feature points to obtain a third transformation relationship, wherein the optimization adopts the following objective function:

[0014] in, For the preset projection function, Let j be the coordinates of the j-th remaining feature point in the car entrance reference coordinate system. Let K be the actual pixel coordinates of the j-th remaining feature point, and K be the camera's intrinsic parameters. This represents the candidate transformation relationship.

[0015] By using the objective function as a benchmark and performing multiple iterations, a more accurate third transformation relationship between the car entrance reference coordinate system and the camera coordinate system can be determined. Furthermore, this scheme uses only accurate feature points to calculate the total weight projection error, eliminating abnormal feature points caused by occlusion, shadows, or misidentification. This results in a more accurate third transformation relationship and ensures precise positioning.

[0016] In one optional implementation, determining the heading angle of the carriage entrance reference coordinate system relative to the mobile device coordinate system according to the second transformation relationship includes: The target edge is determined from the structural features. The target edge includes a first feature point and a second feature point, which are the two endpoints of the target edge. Based on the second transformation relationship, determine the coordinates of the first feature point and the second feature point in the mobile device coordinate system; Determine the orientation angle of the target edge based on the coordinates of the first and second feature points in the mobile device coordinate system; The heading angle is determined based on the direction angle of the target side and the preset angle between the target side and the heading direction of the carriage entrance.

[0017] Since there may be a fixed geometric relationship between different structural edges and the carriage entrance heading direction, such as the different angles between the bottom edge of the carriage entrance, the side edge of the door frame, the edge of the floor plate and the carriage heading direction, a preset angle can be introduced for correction to avoid directly mistaking the direction of the structural edge as the carriage heading direction.

[0018] In one alternative implementation, determining the target edge from structural features includes: Robust fitting is performed on multiple candidate structural edges identified from the target image to obtain fitting results corresponding to each candidate structural edge. The fitting results include at least one of fitting error, line length, structural type confidence, and edge integrity. Based on the fitting results, the scores corresponding to the multiple candidate structure edges are determined respectively; Candidate structural edges whose scores meet preset conditions are identified as target edges.

[0019] In this way, by obtaining the fitting result through the fitting operation, the score can be determined based on the fitting result, and the candidate structural edge that meets the preset conditions can be selected as the target edge. This allows for the use of higher quality structural edges when determining the heading angle in the subsequent process, which is more accurate and avoids the problem of misidentification of the heading angle caused by the lack of information due to poor quality structural edges such as cargo edges and vehicle outlines.

[0020] In a second aspect, the present invention provides a positioning device for a mobile device, the device comprising: The acquisition module is used to acquire, in the current cycle, the target image captured by the camera, the motion information of the mobile device, and the size information of the carriage, wherein the camera is mounted on the mobile device; The recognition module is used to identify the structural features of the carriage entrance, wherein the structural features include multiple feature points; The extraction module is used to extract the actual pixel coordinates of multiple feature points from the target image; The module is used to construct a reference coordinate system for the carriage entrance with the reference point at the carriage entrance as the origin, and to determine the carriage coordinates of multiple feature points in the reference coordinate system based on the size information, wherein the reference point at the carriage entrance is the center point of the edge of the carriage entrance floor. Based on the first transformation relationship between the preset camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of multiple feature points, and the camera's intrinsic parameters, the module determines the second transformation relationship between the reference coordinate system at the carriage entrance and the mobile device coordinate system. Based on the second transformation relationship, the module determines the position of the reference point at the carriage entrance in the mobile device coordinate system, and also determines the heading angle of the reference coordinate system at the carriage entrance relative to the mobile device coordinate system. The module is used to construct the visual observations for the current cycle based on the position and heading angle of the car entrance reference point; The fusion module is used to fuse visual observations and motion information of the current cycle to obtain target pose information, which is used to characterize the relative pose of the carriage entrance reference coordinate system with respect to the mobile device coordinate system.

[0021] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the positioning method of the mobile device described in the first aspect or any corresponding embodiment thereof.

[0022] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the positioning method of a mobile device according to the first aspect or any corresponding embodiment thereof.

[0023] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the positioning method of a mobile device according to the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of a first method for locating a mobile device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a second process for locating a mobile device according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a positioning device for a mobile device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0028] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0029] The embodiments of this application can be applied to mobile devices, such as... Figure 1 As shown, the mobile device may include a camera and electronic devices. The camera, which can be a monocular camera, a binocular camera, or a depth camera, is used to capture images and transmits the captured images to the electronic devices. The electronic devices can be used to determine the target pose information based on the images captured by the camera, the motion information of the mobile device, and the size information of the vehicle compartment. The mobile device can be used for loading and unloading goods in the vehicle compartment; for example, the mobile device may be an unmanned forklift, an automated guided vehicle (AGV), or a shelf-mounted robot. The electronic devices may be domain controllers, onboard computing units, industrial control computers, or embedded controllers within the mobile device.

[0030] According to an embodiment of the present invention, a positioning method for a mobile device is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] This embodiment provides a method for locating a mobile device, which can be executed by an electronic device. Figure 2 This is a flowchart of a mobile device positioning method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: In the current cycle, acquire the target image captured by the camera, the motion information of the mobile device, and the size information of the carriage.

[0032] The camera is mounted on a mobile device. The target image can be an image of the vehicle's cargo compartment entrance taken by the camera in the platform area. Motion information can include at least one of speed, angular velocity, wheel speed, odometer information, and inertial measurement information. Cargo compartment dimensional information includes at least one of the following: car compartment width, car compartment height, door frame dimensions, floor edge position, corner fitting position, and car compartment entrance structural dimensions.

[0033] Specifically, after the vehicles to be loaded / unloaded stop at the platform, the camera on the mobile device can periodically take pictures of the vehicle's entrance, and the electronic equipment can acquire the target images captured by the camera. The mobile device can also be equipped with wheel speed meters and inertial measurement units (IMUs). The wheel speed meters can measure the speed of the mobile device, and the inertial measurement units can measure the angular velocity, acceleration, or attitude changes of the mobile device. The electronic equipment can periodically collect the speed of the mobile device from the wheel speed meters and periodically collect the angular velocity of the mobile device from the inertial measurement units.

[0034] In addition, electronic devices can also acquire the dimensions of the carriage. These dimensions can be pre-stored in the electronic devices or provided by the dispatching system, vehicle identification system, or manual input. For example, for a specific vehicle model or carriage type, corresponding dimensions such as carriage width, height, door frame width, door frame height, and floor edge position can be pre-configured.

[0035] Step S202: Identify the structural features of the carriage entrance in the target image, wherein the structural features may include multiple feature points.

[0036] Specifically, this embodiment selects the structural features of the vehicle's entrance for location. After acquiring the target image of the vehicle's entrance, the electronic device can use a preset visual algorithm to identify the structural features of the vehicle's entrance from the target image. For example, the visual algorithm may include one or more algorithms such as edge detection, line detection, corner detection, and semantic segmentation. The structural features may include the vehicle entrance door frame edge, vehicle entrance floor edge, vehicle entrance bottom edge, vehicle side profile edge, corner pieces, corner piece lines, door frame corner points, and floor edge endpoints. Feature points can be sampling points on structural edges, or intersections, corner points, endpoints, or key points obtained from semantic segmentation.

[0037] Optionally, after acquiring the target image, the electronic device can first perform image enhancement processing to obtain a processed target image. Then, a visual algorithm can be used to identify structural features from the processed target image. This can improve image quality and enhance the accuracy of feature point recognition.

[0038] In one alternative implementation, the target image can be obtained through multi-frame accumulation image processing techniques, which can improve image brightness in low-light environments.

[0039] In the process of identifying the structural features of the carriage entrance, electronic devices can specifically employ the following methods: Edge detection and line detection are used to extract points on various structural edges of the carriage from the target image. For example, structural edges can be the two sides of the carriage entrance door frame, the lower edge of the floor, the lateral contour edges of the carriage, and the lines connecting corner pieces. Furthermore, the intersection of two edges can be identified as corner points. Alternatively, corner point detection can be used to directly identify each corner point from the target image. Alternatively, semantic segmentation algorithms can be used to determine feature points of different semantic types, such as the side edge of the carriage entrance, the bottom edge of the carriage entrance, etc.

[0040] Step S203: Extract the actual pixel coordinates of multiple feature points from the target image.

[0041] Specifically, after identifying the structural features of the carriage entrance, the electronic equipment can extract the actual pixel coordinates of each feature point in the pixel coordinate system of the target image. For the j-th feature point, its actual pixel coordinates can be expressed as: p j =(u j ,v j ), where u j and v j These represent the horizontal and vertical pixel coordinates of the j-th feature point in the target image, respectively. The actual pixel coordinates can be obtained through edge detection, corner detection, semantic segmentation post-processing, line fitting, or keypoint regression.

[0042] Step S204: Construct a reference coordinate system for the car entrance with the reference point at the car entrance as the origin, and determine the car coordinates of multiple feature points in the reference coordinate system based on the dimensional information. The reference point for the car entrance is the center point of the edge of the car entrance floor plate.

[0043] The center point of the carriage entrance floor edge can be understood as the center point of the carriage entrance floor edge in the width direction. This point can serve as a reference position for mobile equipment to perform alignment, carriage entry, and loading / unloading operations. In one example, the carriage entrance reference coordinate system can be represented as coordinate system E. The carriage entrance reference coordinate system has the center point of the carriage entrance floor edge as its origin, X... E The axis points inwards towards the interior of the carriage, Y E Along the width direction of the carriage entrance, Z E The axis points upwards from the carriage. Therefore, the coordinates of the carriage entrance reference point in the carriage entrance reference coordinate system are: O E =(0,0,0).

[0044] Specifically, electronic devices can determine the coordinates of each feature point in the car entrance reference coordinate system based on the car's size information, such as the car's width, height, door frame size, floor edge position, and corner piece position.

[0045] For example, if the width of the carriage is W and the height is H, and the center point of the bottom edge of the carriage entrance is the origin of the carriage entrance reference coordinate system, with the X-axis pointing towards the inside of the carriage, the Y-axis pointing towards the right side of the carriage, and the Z-axis pointing towards the top of the carriage, then in this coordinate system, for the four corner points P1 (top left), P2 (top right), P3 (bottom left), and P4 (bottom right) of the plane containing the carriage entrance, the carriage coordinates of these four corner points in the carriage entrance reference coordinate system are as follows: P1 = (0, -W / 2, H) P2 = (0, W / 2, H) P3 = (0, -W / 2, 0) P4 = (0, W / 2, 0) In another example, if the structural feature consists of multiple sampling points on the edge of the carriage entrance floor, the coordinates of each sampling point in the carriage entrance reference coordinate system can be determined based on the carriage entrance width W and the relative position of each sampling point on the edge of the floor. For example, if the edge of the floor is located at X... E =0 and Z E On the straight line where y = 0, any point on the edge of the base plate can be represented as: P = (0, y, 0). Where y is the coordinate of the point in the width direction of the carriage entrance.

[0046] The above method can establish the correspondence between the actual pixel coordinates of feature points in the target image and the coordinates of the carriage in the reference coordinate system of the carriage entrance.

[0047] Step S205: Based on the preset first transformation relationship between the camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of multiple feature points, and the camera's intrinsic parameters, determine the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system.

[0048] Specifically, since the carriage coordinates and actual pixel coordinates can represent the relationship between the carriage entrance reference coordinate system and the image plane, and the camera's intrinsic parameters can represent the relationship between the image plane and the camera coordinate system, and the first transformation relationship is the relationship between the camera coordinate system and the mobile device coordinate system, the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system can be determined based on the first transformation relationship, the carriage coordinates and actual pixel coordinates of multiple feature points, and the camera's intrinsic parameters.

[0049] Step S206: Determine the position of the car entrance reference point in the mobile device coordinate system according to the second transformation relationship, and determine the heading angle of the car entrance reference coordinate system relative to the mobile device coordinate system according to the second transformation relationship.

[0050] Since the reference point at the carriage entrance is the origin of the carriage entrance reference coordinate system, its position in the mobile device coordinate system can be determined according to the second transformation relationship. Specifically, the electronic device can use the second transformation relationship to perform a coordinate system transformation operation on the coordinates of the carriage entrance reference point to obtain its position in the mobile device coordinate system. Because the second transformation relationship indicates the transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system, the heading angle of the carriage entrance reference coordinate system relative to the mobile device coordinate system can be determined based on this relationship.

[0051] Step S207: Based on the position and heading angle of the car entrance reference point, construct the visual observation for the current cycle.

[0052] Specifically, the visual observations for the current cycle can include the position of the carriage entrance reference point in the mobile device coordinate system and the heading angle of the carriage entrance reference coordinate system relative to the mobile device coordinate system. For example, constructing the visual observations for the current cycle can be represented as z k = (x k y k θ k ); where x k This indicates the position of the current cycle's carriage entrance reference point in the direction of the mobile device's travel, y k θ represents the position of the current cycle's carriage entrance reference point in the lateral direction of the mobile device. k This indicates the heading angle of the current cycle's carriage entrance reference coordinate system relative to the mobile device's coordinate system. Visual observations can simultaneously indicate the longitudinal distance, lateral offset, and heading deviation of the carriage entrance relative to the mobile device, facilitating positioning control, path planning, and carriage entry operations for the mobile device.

[0053] Step S208: The visual observations and motion information of the current cycle are fused to obtain the target pose information. The target pose information can be used to characterize the relative pose of the carriage entrance reference coordinate system with respect to the mobile device coordinate system.

[0054] Specifically, the electronic device can input the visual observations and motion information of the current cycle into a pre-built Extended Kalman Filter (EKF) algorithm. The EKF algorithm fuses the observations and motion information and performs filtering to obtain the target pose information of the current cycle. The target pose information is used to characterize the relative pose of the car entrance reference coordinate system with respect to the mobile device coordinate system.

[0055] The target image may be affected by factors such as changes in lighting, occlusion, reflection, and missing edges, which may cause jitter or error in the single-frame visual positioning result. Motion information can indicate the motion state of the mobile device and can be collected by the sensors on the mobile device itself. Therefore, this solution fuses the visual observations of the current period with motion information, which can reduce the impact of poor image quality on positioning accuracy.

[0056] The mobile device positioning method provided in this embodiment acquires, in the current cycle, a target image captured by a camera mounted on the mobile device, motion information of the mobile device, and size information of the carriage. It identifies the structural features of the carriage entrance in the target image and extracts the actual pixel coordinates of feature points within these structural features. Then, it constructs a reference coordinate system for the carriage entrance based on the size information and determines the carriage coordinates of the structural features within this reference coordinate system. Further, it determines a second transformation relationship between the reference coordinate system for the carriage entrance and the mobile device coordinate system using a first transformation relationship between the camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of multiple feature points, and the camera's intrinsic parameters. This second transformation relationship is then used to determine the position of the carriage entrance reference point in the mobile device coordinate system and the heading angle of the reference coordinate system for the carriage entrance relative to the mobile device coordinate system, and a visual observation is constructed based on the position and heading angle. Finally, the visual observation and motion information of the current cycle are fused to obtain the target pose information.

[0057] Because the relative position between the mobile device and the train carriage changes continuously during the movement of the mobile device, and the characteristics of the carriage captured by the camera differ at different relative positions, this solution utilizes the structural features of the carriage entrance itself for positioning. It determines the position of the carriage entrance reference point and the heading angle of the carriage entrance reference coordinate system, indicating the relative positional relationship between the carriage entrance reference coordinate system and the mobile device. Furthermore, since single-frame images may not be accurate enough, this solution fuses motion information indicating the mobile device's motion state with visual observations to obtain the target pose information of the carriage entrance reference coordinate system relative to the mobile device's coordinate system. Even if the vehicle to be loaded / unloaded is not strictly parked in the preset position, the mobile device can still locate itself based on the real-time position and orientation of the carriage entrance by recognizing its structural features. This ensures that the mobile device can adapt to vehicle parking deviations without the need for other auxiliary facilities, significantly reducing costs and improving the adaptability of the mobile device in platform loading / unloading scenarios.

[0058] In some alternative implementations, step S208 may take the following specific steps: Step 1: Based on the motion information and the target pose information of the previous cycle, predict the initial relative pose information of the current cycle.

[0059] Step 2: Obtain the quality score of the target image.

[0060] Step 3: Based on the quality score, adjust the preset visual observation noise covariance to obtain the visual observation noise covariance for the current period.

[0061] Among them, the lower the quality score, the greater the visual observation noise covariance in the current period.

[0062] Step 4: Determine the target pose information for the current period based on the visual observations of the current period, the visual observation noise covariance of the current period, and the initial relative pose information of the current period.

[0063] In one alternative implementation, the electronic device may employ an extended Kalman filter algorithm to fuse visual observations and motion information. Specifically, the initial relative pose information for the current cycle can be predicted based on the motion information and the target pose information from the previous cycle; then, the initial relative pose information is updated based on the visual observations for the current cycle to obtain the target pose information for the current cycle.

[0064] Specifically, in step one, within the current cycle, motion information and target pose information from the previous cycle can be input into a preset motion function to obtain the initial relative pose information for the current cycle output by the preset motion function. The preset motion function can be a mathematical expression describing the changes in an object's position, velocity, acceleration, etc., over time, used to analyze and predict the object's motion state.

[0065] In step two, the electronic device can evaluate the quality of the target image and obtain a quality score.

[0066] Optionally, the electronic device can perform a scoring operation on the target image in at least one dimension to obtain at least one sub-score. Then, it can perform a weighted calculation on the sub-scores in at least one dimension to obtain a quality score. Multiple dimensions may include information such as the number of corner points, corner confidence (directly given by the visual algorithm), the integrity of the carriage entrance edge, reprojection error, and the average image brightness. For each dimension, a scoring mapping relationship (which can be a table or a mathematical expression) can be set. Taking the target dimension (any dimension) as an example, the corresponding sub-score can be determined based on the data of the target dimension and the scoring mapping relationship. For example, for corner confidence, the scoring mapping relationship can be a formula for calculating the mean; correspondingly, the mean of at least one corner confidence can be determined as the sub-score for that dimension.

[0067] In step three, the electronic device can adjust the preset visual observation noise covariance using the aforementioned quality score to obtain the visual observation noise covariance for the current period. The lower the quality score, the larger the visual observation noise covariance for the current period. For example, the ratio between the preset visual observation noise covariance and the quality score can be determined as the visual observation noise covariance for the current period.

[0068] In step four, the electronic device can determine the observation residual as the difference between the visual observation measurement and the initial relative pose information of the current cycle. Then, using the Kalman gain calculation formula, the visual observation noise covariance and motion prediction noise covariance of the current cycle are calculated to obtain the Kalman gain. The motion prediction noise covariance, like the pose information, is updated in each cycle and can be calculated using existing methods, which will not be elaborated here. The electronic device can calculate the product of the Kalman gain and the observation residual, and add this product to the initial relative pose information of the current cycle to obtain the target pose information of the current cycle.

[0069] For example, the target pose information from the previous cycle is represented as:

[0070] in, This indicates the position of the car entrance reference point in the direction of travel of the mobile equipment during the previous cycle. This indicates the position of the car entrance reference point in the lateral direction of the mobile equipment during the previous cycle. This represents the heading angle of the car entrance reference coordinate system relative to the mobile device coordinate system in the previous cycle.

[0071] The initial relative pose information of the current cycle can be represented as .

[0072] in, This represents the motion information for the current cycle, where f represents the preset motion function. Motion information It may include at least one of the following: the speed, angular velocity, wheel speed, odometer information, and inertial measurement information of the mobile device.

[0073] The relationship between the visual observations of the current cycle and the initial relative pose information can be expressed as:

[0074] in, Here, h represents the visual observations for the current period, and h is the preset observation model. This provides the initial relative pose information for the current cycle. This refers to visual observation noise. The covariance can be expressed as the visual observation noise covariance. .

[0075] The observation residual can be expressed as .

[0076] Electronic devices can update the initial relative pose information based on the observation residual, visual observation noise covariance, and motion prediction noise covariance to obtain the target pose information for the current cycle.

[0077] In one alternative implementation, the visual observation noise covariance can be a diagonal matrix, which can be expressed as follows:

[0078] in, Represents the visual observation noise covariance. This represents the variance of visual observations in the direction of travel of the mobile device. This represents the variance of visual observations in the lateral direction of the mobile device. The variance of visual observations represents the heading angle.

[0079] In some optional implementations, in step S205 above, determining the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system based on the preset first transformation relationship between the camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of multiple feature points, and the camera's intrinsic parameters may include the following specific steps: Step 1: Based on the carriage coordinates and actual pixel coordinates of multiple feature points, as well as the camera's intrinsic parameters, determine the third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system.

[0080] Step two: Determine the second transformation relationship based on the first and third transformation relationships.

[0081] Specifically, the camera's intrinsic parameters can be used to indicate the transformation relationship between the camera coordinate system and the image plane, and the relationship between the carriage coordinates and the pixel coordinates can indicate the transformation relationship between the carriage entrance reference coordinate system and the image plane. Therefore, the electronic device can input the carriage coordinates and actual pixel coordinates of multiple feature points in the carriage entrance reference coordinate system, as well as the camera's intrinsic parameters, into a pre-constructed Perspective-n-Point (PnP) algorithm (or homography algorithm). After the PnP algorithm (homography algorithm) solves the problem, the third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system is obtained.

[0082] Since the first transformation relationship can indicate the first transformation relationship between the camera coordinate system and the mobile device coordinate system, and the third transformation relationship can indicate the transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system, the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system can be determined by calculating the first and third transformation relationships.

[0083] For ease of explanation, we can denote the reference coordinate system at the carriage entrance as E, the camera coordinate system as C, and the mobile device coordinate system as B. The first transformation relationship between the camera coordinate system and the mobile device coordinate system can be denoted as: This is used to represent the transformation relationship from the camera coordinate system to the mobile device coordinate system. The third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system can be denoted as: This represents the transformation relationship from the carriage entrance reference coordinate system to the camera coordinate system. The second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system can be denoted as... , is used to represent the transformation relationship from the reference coordinate system at the entrance of the carriage to the coordinate system of the mobile device.

[0084] In one optional implementation, the electronic device can determine a third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system based on the carriage coordinates, actual pixel coordinates, and camera intrinsic parameters of multiple feature points. For example, electronic devices can use the perspective n-point problem algorithm, namely the PnP algorithm, to solve the third transformation relationship based on the correspondence between the three-dimensional carriage coordinates and the two-dimensional pixel coordinates.

[0085] Third transformation relationship It can include rotation and translation components, represented as:

[0086] in, This indicates the rotation relationship between the reference coordinate system at the carriage entrance and the camera coordinate system. This indicates the position of the origin of the reference coordinate system at the entrance of the carriage in the camera coordinate system.

[0087] The first transformation relationship between the camera coordinate system and the mobile device coordinate system This can be predetermined through camera extrinsic parameter calibration. First transformation relationship. It can be represented as:

[0088] in, This indicates the rotation relationship between the camera coordinate system and the mobile device coordinate system. This indicates the position of the camera coordinate system origin in the mobile device coordinate system.

[0089] After obtaining the first transformation relationship and the third transformation relationship Then, the second transformation relationship between the car entrance reference coordinate system and the mobile device coordinate system can be determined. For example, when the conversion directions are the same, the following expression can be used: in, This indicates the transformation relationship between the reference coordinate system at the carriage entrance and the coordinate system of the mobile device.

[0090] Second transformation relationship It can be represented as .

[0091] in, This indicates the rotational relationship between the reference coordinate system at the carriage entrance and the coordinate system of the moving equipment. This indicates the position of the origin of the car entrance reference coordinate system in the mobile device coordinate system. Since the car entrance reference coordinate system takes the car entrance reference point as its origin, therefore... It can be used to represent the position of the car entrance reference point in the mobile device coordinate system.

[0092] In some optional implementations, in step one of step S205 above, the electronic device may use the following specific steps to determine the third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system based on the carriage coordinates and pixel coordinates of multiple feature points and the camera's intrinsic parameters: Step 1: Obtain the initial transformation relationship between the car entrance reference coordinate system and the camera coordinate system, and use the initial transformation relationship as the initial value of the candidate transformation relationship.

[0093] The initial transformation relationship can be determined based on the third transformation relationship of the previous cycle, the second transformation relationship of the previous cycle, the motion prediction results of the mobile device, the coarse positioning results of the carriage entrance, or the preset initial value.

[0094] Step 2: Based on the current candidate transformation relationship, the camera's intrinsic parameters, and the carriage coordinates of multiple feature points, determine the predicted pixel coordinates of multiple feature points respectively.

[0095] Step 3: Determine the reprojection error corresponding to each feature point based on the predicted and actual pixel coordinates of the multiple feature points.

[0096] Step 4: Remove feature points whose reprojection error is greater than the preset error threshold to obtain multiple remaining feature points.

[0097] Step 5: Based on the total reprojection error corresponding to multiple remaining feature points, optimize the candidate transformation relationship to obtain the third transformation relationship.

[0098] Specifically, in step 1, when the current iteration is the first iteration, a preset initial transformation relationship can be used as a candidate transformation relationship for the current iteration and participate in subsequent iteration calculations. When the current iteration is not the first iteration, the candidate transformation relationship can be the transformation relationship obtained through the adjustment operation of the previous iteration.

[0099] In step 2, in any iteration, for any feature point, the electronic device can input the candidate transformation relationship corresponding to the current iteration, the camera's intrinsic parameters, and the carriage coordinates of the feature point into the pre-constructed projection function to obtain the predicted pixel coordinates of the feature point output by the projection function.

[0100] In step 3, after determining the predicted pixel coordinates of each feature point, the electronic device can determine the reprojection error between the predicted pixel coordinates and the actual pixel coordinates of each feature point.

[0101] In step 4, it can be determined whether the reprojection error of each feature point is greater than a preset error threshold. If it is, the corresponding feature point can be removed; otherwise, the corresponding feature point can be retained. In this way, multiple remaining feature points can be obtained.

[0102] In step 5, the reprojection errors of each remaining feature point are summed to obtain the total reprojection error. Then, using the objective function as a benchmark, the candidate transformation relationship corresponding to the current iteration is optimized to obtain the third transformation relationship. For example, based on the total reprojection error, the Jacobian matrix is ​​calculated, and the candidate transformation relationship is adjusted using the Jacobian matrix to obtain a new candidate transformation relationship (which can be used as the candidate transformation relationship for the next iteration). This continues until any iteration meets the iteration stopping condition (e.g., the total reprojection error is less than a preset error threshold, the number of iterations reaches a preset number, and the change in total reprojection error between two adjacent iterations is less than a preset change threshold). Finally, the candidate transformation relationship of the last iteration is determined as the final third transformation relationship.

[0103] For example, the objective function can be expressed as follows:

[0104] in, For the preset projection function, Let j be the coordinates of the j-th remaining feature point in the car entrance reference coordinate system. Let K be the actual pixel coordinates of the j-th remaining feature point, and K be the camera's intrinsic parameters. This represents the candidate transformation relationship.

[0105] When the iteration stopping condition is met, the candidate transformation relationship obtained in the last iteration is determined as the third transformation relationship. The iteration stopping condition may include at least one of the following: the total reprojection error is less than a preset error threshold, the number of iterations reaches a preset number, and the change in the total reprojection error between two adjacent iterations is less than a preset change threshold.

[0106] In some optional implementations, in step S206 above, the following specific steps can be used to determine the heading angle of the carriage entrance reference coordinate system relative to the mobile device coordinate system based on the second transformation relationship: Step 1: Determine the target edge from the structural features. The target edge includes a first feature point and a second feature point, where the first feature point and the second feature point are the two endpoints of the target edge.

[0107] Step 2: Based on the second transformation relationship, determine the coordinates of the first feature point and the second feature point in the mobile device coordinate system, and determine the direction angle of the target edge.

[0108] Step 3: Determine the heading angle based on the direction angle of the target side and the preset angle between the target side and the heading direction of the carriage entrance.

[0109] Specifically, in step one, the second transformation relationship can be used to perform coordinate system transformation operations on the carriage coordinates of the first feature point and the second feature point respectively, so as to obtain the coordinates of the first feature point in the mobile device coordinate system and the coordinates of the second feature point in the mobile device coordinate system.

[0110] In step two, the coordinates of the aforementioned feature points in the mobile device coordinate system may include lateral coordinates and travel direction coordinates. The electronic device can calculate the first difference between the lateral coordinates of the first feature point and the second feature point, calculate the second difference between the travel direction coordinates of the first feature point and the second feature point, and then calculate the direction angle of the target side based on the second difference and the first difference.

[0111] In step three, the sum of the direction angle of the target side and the preset included angle can be determined as the heading angle.

[0112] For example, the coordinates of the first feature point in the mobile device coordinate system are: ; The coordinates of the second feature point in the mobile device coordinate system are: ; in, and The coordinates are in the direction the mobile device is traveling. and These are the coordinates in the lateral direction of the mobile device.

[0113] When the target edge points from the first feature point to the second feature point, the change in coordinates along the direction of travel of the mobile device can be expressed as: When the target edge points from the first feature point to the second feature point, the change in coordinates in the lateral direction of the mobile device can be expressed as: .

[0114] The orientation angle of the target edge in the mobile device coordinate system can be expressed as:

[0115]

[0116]

[0117]

[0118] in, Let the direction angle of the target edge be in the mobile device coordinate system. Let these be the coordinates of the first feature point in the mobile device coordinate system. Here are the coordinates of the second feature point in the mobile device's coordinate system, where the x-direction represents the mobile device's travel direction or longitudinal direction, and the y-direction represents the mobile device's lateral direction or transverse direction. This is the change in coordinates of the target edge in the direction of travel of the mobile device when it points from the first feature point to the second feature point, which is also the second difference mentioned above. This is the change in coordinates of the target edge in the lateral direction of the mobile device when it points from the first feature point to the second feature point, which is also the first difference mentioned above. Let be the heading angle of the reference coordinate system at the carriage entrance relative to the coordinate system of the mobile device. The preset angle is the angle between the target side and the heading direction of the reference coordinate system at the entrance of the carriage. The preset angle is determined according to the structural type of the target side (e.g., the side or bottom edge of the door frame). For example, the angle corresponding to the structural type can be obtained as the preset angle based on the structural type.

[0119] The preset included angle can be determined based on the structural type of the target edge. For example, if the target edge is the edge of the carriage entrance floor, then the target edge is usually along the width direction of the carriage entrance, and the included angle between it and the carriage entrance heading direction can be approximately [missing information]. or If the target side is the side of the carriage entrance, the angle between the target side and the heading direction of the carriage entrance can be preset according to the carriage structure. By introducing a preset angle, the direction of the target side can be avoided from being mistakenly identified as the heading direction of the carriage entrance, thereby improving the accuracy of the heading angle determination.

[0120] In some optional implementations, before step one in step S206 above, the operation of selecting the target edge can be performed first, and the specific steps can be as follows: Step 1: Robustly fit the multiple candidate structural edges identified from the target image to obtain the fitting results corresponding to each candidate structural edge.

[0121] The fitting result includes at least one of the following: fitting error, line length, structure type confidence, and edge integrity. Fitting error can be the average vertical distance from the supporting pixels (also called inliers) of the candidate structure edge to the fitted line. Line length can be the physical length of the candidate structure edge in the mobile device coordinate system. Structure type confidence can be the probability that the candidate structure edge belongs to the true structure edge of the carriage entrance. Edge integrity can be the degree of continuous coverage of the edge response within the inlier span of the candidate structure edge.

[0122] Step 2: Based on the fitting results, determine the scores corresponding to the multiple candidate structural edges.

[0123] Step 3: Select the candidate structural edges whose scores meet the preset conditions as the target edges.

[0124] Specifically, in step one, the electronic device can use a multi-line random sampling consensus (RANSAC) algorithm to robustly fit multiple candidate structural edges (identified by visual algorithms) identified from the target image, obtaining fitting results for each candidate structural edge. For each candidate structural edge, a score can be determined based on its fitting result. For example, if only one fitting result is available, the score corresponding to the fitting result can be determined based on the fitting result and the mapping relationship between the fitting result and the score. If multiple fitting results are available, the scores corresponding to each fitting result can be determined first, and then the multiple scores can be weighted and summed. Finally, candidate structural edges whose scores meet preset conditions can be determined as target edges; for example, the preset condition could be a score greater than a preset score threshold.

[0125] In some alternative implementations, the camera can be a stereo camera. For the left and right images captured by the stereo camera, the electronic device can triangulate the pixel coordinates of the same feature point in both images based on the intrinsic and extrinsic parameters of the stereo camera, obtaining the three-dimensional coordinates of the feature point in the camera coordinate system. The electronic device can then use the triangulated three-dimensional coordinates to assist in solving or verifying the third transformation relationship.

[0126] In some alternative implementations, the mobile device can also be equipped with a LiDAR. The electronic device can acquire point cloud data collected by the LiDAR and match the structural points in the point cloud data with feature points in the target image. The electronic device can transform the coordinates of the feature points in the point cloud data from the LiDAR coordinate system to the camera coordinate system, and then project them onto the image plane to obtain the corresponding predicted pixel coordinates. If the deviation between the predicted pixel coordinates and the actual pixel coordinates is small, the feature point can be retained; if the deviation is large, the feature point can be discarded. In this way, point cloud data can be used to assist in the removal of misidentified feature points, improving the accuracy of solving the third transformation relationship.

[0127] It should be noted that this embodiment can complete the positioning using only the camera and motion information, or it can be further combined with a binocular camera, depth camera or lidar as auxiliary information. This embodiment of the invention does not limit this.

[0128] The following example illustrates the positioning method for the mobile device described above. The mobile device is an unmanned forklift.

[0129] like Figure 3 As shown, the unmanned forklift is equipped with a camera, wheel speedometer, and inertial measurement unit. The camera is used to capture images of the truck bed entrance of vehicles waiting to be loaded or unloaded in the platform area, while the wheel speedometer and inertial measurement unit are used to collect motion information of the unmanned forklift.

[0130] First, the electronic equipment can acquire target images from the camera mounted on the unmanned forklift. Then, it uses visual algorithms to identify the structural features of the truck entrance (i.e., the aforementioned feature points) from the target image and extracts the actual pixel coordinates of multiple feature points. Next, the electronic equipment constructs a reference coordinate system for the truck entrance with the center point of the truck entrance floor edge as the origin. Combining this with the truck's dimensional information, such as width, height, door frame dimensions, and floor edge position, it determines the truck's coordinates within the reference coordinate system for the multiple feature points.

[0131] Furthermore, the electronic device employs the PnP algorithm to determine the third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system based on the carriage coordinates, actual pixel coordinates, and camera intrinsic parameters of multiple feature points. The electronic device then determines the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system based on the first and third transformation relationships between the camera coordinate system and the mobile device coordinate system.

[0132] Furthermore, the electronic equipment determines the position of the car entrance reference point in the mobile device coordinate system based on the second transformation relationship, and determines the heading angle of the car entrance reference coordinate system relative to the mobile device coordinate system. The electronic equipment can then construct the position and heading angle into a visual observation for the current cycle.

[0133] In addition, the electronic equipment acquires motion information from the wheel speedometer and IMU on the unmanned forklift, and uses the EKF algorithm to fuse and filter the visual observation results (including the position of the truck entrance reference point in the mobile device coordinate system and the heading angle of the truck entrance reference coordinate system relative to the mobile device coordinate system) and motion information to obtain target pose information. The target pose information is used to characterize the relative pose of the truck entrance reference coordinate system relative to the mobile device coordinate system. Based on the target pose information, the unmanned forklift determines its lateral deviation, longitudinal distance, and heading deviation from the truck entrance, and performs alignment control, path planning, and truck entry operations based on these deviations. Even when there is a deviation in the parking position of the vehicle to be loaded or unloaded, the unmanned forklift can still locate itself based on the actual position and orientation of the truck entrance, thereby improving the adaptability of platform loading and unloading operations.

[0134] Due to the open space of the platform area, mobile devices often require auxiliary facilities for positioning, resulting in high maintenance costs. This solution eliminates the need for such auxiliary facilities, thus reducing maintenance costs. The unmanned forklift uses the entrance of the vehicle to be loaded / unloaded as a reference point. Even if the vehicle's parking position deviates, it can still achieve positioning based on the actual location and orientation of the entrance. Furthermore, this solution reduces the impact of lighting conditions through image enhancement processing and motion information fusion.

[0135] This embodiment also provides a positioning device for a mobile device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0136] This embodiment provides a positioning device for a mobile device, such as... Figure 4 As shown, it includes: The acquisition module 410 is used to acquire, in the current cycle, the target image captured by the camera, the motion information of the mobile device, and the size information of the carriage, wherein the camera is mounted on the mobile device; The recognition module 420 is used to recognize the structural features of the carriage entrance, wherein the structural features include multiple feature points; Extraction module 430 is used to extract the actual pixel coordinates of multiple feature points from the target image; The module 440 is used to construct a reference coordinate system for the carriage entrance with the reference point at the carriage entrance as the origin, and to determine the carriage coordinates of multiple feature points in the reference coordinate system based on the size information, wherein the reference point for the carriage entrance is the center point of the edge of the carriage entrance floor. Based on a preset first transformation relationship between the camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of the multiple feature points, and the camera's intrinsic parameters, a second transformation relationship between the reference coordinate system for the carriage entrance and the mobile device coordinate system is determined. Based on the second transformation relationship and the reference point coordinates, the position of the reference point for the carriage entrance in the mobile device coordinate system is determined, and the heading angle of the reference coordinate system for the carriage entrance relative to the mobile device coordinate system is determined based on the second transformation relationship. Module 450 is used to construct the visual observations for the current period based on the position and heading angle of the car entrance reference point; The fusion module 460 is used to fuse the visual observations and motion information of the current cycle to obtain the target pose information, wherein the target pose information is used to characterize the relative pose of the car entrance reference coordinate system with respect to the mobile device coordinate system.

[0137] In some alternative implementations, the fusion module 460 is specifically used for: Based on motion information and target pose information from the previous cycle, predict the initial relative pose information for the current cycle. Obtain the quality score of the target image; Based on the quality score, the preset visual observation noise covariance is adjusted to obtain the visual observation noise covariance for the current period. The lower the quality score, the larger the visual observation noise covariance for the current period. Based on the visual observations of the current period, the visual observation noise covariance of the current period, and the initial relative pose information of the current period, the target pose information of the current period is determined.

[0138] In some alternative implementations, the determining module 440 is specifically used for: Based on the carriage coordinates and actual pixel coordinates of multiple feature points, as well as the camera's intrinsic parameters, a third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system is determined. Based on the first and third transformation relations, determine the second transformation relation.

[0139] In some alternative implementations, the determining module 440 is specifically used for: Obtain the initial transformation relationship between the car entrance reference coordinate system and the camera coordinate system, and use the initial transformation relationship as the initial value of the candidate transformation relationship; Based on the current candidate transformation relationship, the camera's intrinsic parameters, and the carriage coordinates of multiple feature points, the predicted pixel coordinates of multiple feature points are determined respectively. Based on the predicted and actual pixel coordinates of multiple feature points, determine the reprojection error corresponding to each feature point. Feature points with reprojection errors greater than a preset error threshold are removed to obtain multiple remaining feature points; Based on the total reprojection error corresponding to multiple remaining feature points, the candidate transformation relationship is optimized to obtain the third transformation relationship.

[0140] In some alternative implementations, the optimization employs the following objective function:

[0141] in, For the preset projection function, Let j be the coordinates of the j-th remaining feature point in the car entrance reference coordinate system. Let K be the actual pixel coordinates of the j-th remaining feature point, and K be the camera's intrinsic parameters. This represents the candidate transformation relationship.

[0142] In some alternative implementations, the determining module 440 is specifically used for: The target edge is determined from the structural features. The target edge includes a first feature point and a second feature point, which are the two endpoints of the target edge. Based on the second transformation relationship, determine the coordinates of the first feature point and the second feature point in the mobile device coordinate system; Determine the orientation angle of the target edge based on the coordinates of the first and second feature points in the mobile device coordinate system; The heading angle is determined based on the direction angle of the target side and the preset angle between the target side and the heading direction of the carriage entrance.

[0143] In some alternative implementations, the determining module 440 is further configured to: Robust fitting is performed on multiple candidate structural edges identified from the target image to obtain fitting results corresponding to each candidate structural edge. The fitting results include at least one of fitting error, line length, structural type confidence, and edge integrity. Based on the fitting results, the scores corresponding to the multiple candidate structure edges are determined respectively; Candidate structural edges whose scores meet preset conditions are identified as target edges.

[0144] The positioning device for mobile devices provided in this embodiment of the invention can execute the positioning method for mobile devices provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0145] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0146] The following is a detailed reference. Figure 5 This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in ROM 502 or a program loaded from memory 508 into RAM 503. RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to bus 504; wherein, ROM is a read-only memory and RAM is a random access memory.

[0147] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays, speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0148] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the positioning method of the mobile device according to embodiments of the present invention.

[0149] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0150] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the positioning method of the mobile device shown in the above embodiments is implemented.

[0151] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0152] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A positioning method for a mobile device, characterized in that, The method includes: In the current cycle, the target image captured by the camera, the motion information of the mobile device, and the size information of the carriage are acquired, wherein the camera is mounted on the mobile device; Identify the structural features of the carriage entrance in the target image, wherein the structural features include multiple feature points; Extract the actual pixel coordinates of multiple feature points from the target image; A reference coordinate system for the car entrance is constructed with the reference point of the car entrance as the origin, and the car coordinates of multiple feature points in the reference coordinate system are determined according to the size information, wherein the reference point of the car entrance is the center point of the edge of the car entrance floor plate; Based on the preset first transformation relationship between the camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of multiple feature points, and the intrinsic parameters of the camera, the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system is determined. The position of the car entrance reference point in the mobile device coordinate system is determined according to the second transformation relationship, and the heading angle of the car entrance reference coordinate system relative to the mobile device coordinate system is determined according to the second transformation relationship. Based on the position of the reference point at the carriage entrance and the heading angle, the visual observation of the current period is constructed; The visual observations of the current cycle and the motion information are fused to obtain target pose information, wherein the target pose information is used to characterize the relative pose of the carriage entrance reference coordinate system with respect to the mobile device coordinate system.

2. The method according to claim 1, characterized in that, The visual observations of the current period and the motion information are fused to obtain target pose information, wherein the target pose information is used to characterize the relative pose of the carriage entrance reference coordinate system with respect to the mobile device coordinate system, including: Based on the motion information and the target pose information of the previous cycle before the current cycle, predict the initial relative pose information of the current cycle; Obtain the quality score of the target image; Based on the quality score, the preset visual observation noise covariance is adjusted to obtain the visual observation noise covariance of the current period, wherein the lower the quality score, the larger the visual observation noise covariance of the current period. The target pose information for the current period is determined based on the visual observations of the current period, the visual observation noise covariance of the current period, and the initial relative pose information of the current period.

3. The method according to claim 1 or 2, characterized in that, The step of determining the second transformation relationship between the carriage entrance reference coordinate system and the mobile device coordinate system based on a preset first transformation relationship between the camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of multiple feature points, and the intrinsic parameters of the camera includes: Based on the carriage coordinates and actual pixel coordinates of multiple feature points, as well as the camera's intrinsic parameters, a third transformation relationship is determined between the carriage entrance reference coordinate system and the camera coordinate system. The second conversion relationship is determined based on the first conversion relationship and the third conversion relationship.

4. The method according to claim 3, characterized in that, The step of determining the third transformation relationship between the carriage entrance reference coordinate system and the camera coordinate system based on the carriage coordinates and actual pixel coordinates of multiple feature points, as well as the camera's intrinsic parameters, includes: Obtain the initial transformation relationship between the reference coordinate system of the carriage entrance and the camera coordinate system, and use the initial transformation relationship as the initial value of the candidate transformation relationship; Based on the current candidate transformation relationship, the camera's intrinsic parameters, and the carriage coordinates of multiple feature points, the predicted pixel coordinates of multiple feature points are determined respectively. Based on the predicted pixel coordinates and actual pixel coordinates of multiple feature points, determine the reprojection error corresponding to each of the multiple feature points; Feature points whose reprojection error is greater than a preset error threshold are removed to obtain multiple remaining feature points; Based on the total reprojection error corresponding to the remaining feature points, the candidate transformation relationship is optimized to obtain the third transformation relationship.

5. The method according to claim 4, characterized in that, The candidate transformation relationship is optimized based on the total reprojection error corresponding to multiple remaining feature points to obtain the third transformation relationship. The optimization adopts the following objective function: in, For the preset projection function, Let j be the coordinates of the j-th remaining feature point in the reference coordinate system at the carriage entrance. Let K be the actual pixel coordinates of the j-th remaining feature point, and K be the intrinsic parameter of the camera. The candidate transformation relationship is denoted as .

6. The method according to claim 1 or 2, characterized in that, Determining the heading angle of the carriage entrance reference coordinate system relative to the mobile device coordinate system based on the second transformation relationship includes: The target edge is determined from the structural features, the target edge including a first feature point and a second feature point, the first feature point and the second feature point being two endpoints on the target edge; Based on the second transformation relationship, determine the coordinates of the first feature point and the second feature point in the mobile device coordinate system; The orientation angle of the target edge is determined based on the coordinates of the first feature point and the second feature point in the coordinate system of the mobile device. The heading angle is determined based on the direction angle of the target side and the preset angle between the target side and the heading direction of the carriage entrance.

7. The method according to claim 6, characterized in that, Determining the target edge from the structural features includes: Robust fitting is performed on multiple candidate structural edges identified from the target image to obtain fitting results corresponding to each candidate structural edge. The fitting results include at least one of fitting error, line length, structural type confidence, and edge integrity. Based on the fitting results, the scores corresponding to the multiple candidate structure edges are determined respectively; Candidate structural edges whose scores meet preset conditions are identified as target edges.

8. A positioning device for a mobile device, characterized in that, include: The acquisition module is used to acquire, in the current period, the target image captured by the camera, the motion information of the mobile device, and the size information of the carriage, wherein the camera is mounted on the mobile device; The identification module is used to identify the structural features of the carriage entrance, wherein the structural features include multiple feature points; An extraction module is used to extract the actual pixel coordinates of multiple feature points from the target image; The module is used to construct a reference coordinate system for the carriage entrance with the reference point at the carriage entrance as the origin, and to determine the carriage coordinates of multiple feature points in the reference coordinate system based on the size information, wherein the reference point at the carriage entrance is the center point of the edge of the carriage entrance floor; based on a preset first transformation relationship between the camera coordinate system and the mobile device coordinate system, the carriage coordinates and actual pixel coordinates of the multiple feature points, and the intrinsic parameters of the camera, to determine a second transformation relationship between the reference coordinate system at the carriage entrance and the mobile device coordinate system; based on the second transformation relationship, to determine the position of the reference point at the carriage entrance in the mobile device coordinate system, and based on the second transformation relationship, to determine the heading angle of the reference coordinate system at the carriage entrance relative to the mobile device coordinate system; A construction module is used to construct the visual observations for the current period based on the position of the reference point at the carriage entrance and the heading angle; The fusion module is used to fuse the visual observations of the current cycle and the motion information to obtain target pose information, wherein the target pose information is used to characterize the relative pose of the carriage entrance reference coordinate system with respect to the mobile device coordinate system.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the positioning method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the positioning method of the mobile device according to any one of claims 1 to 7.

Images