A charging port positioning method and related apparatus

Abstract

The embodiment of the application discloses a charging port positioning method, which comprises the following steps: acquiring a plurality of first point clouds, a plurality of point cloud sets and the positional relationship between each point cloud set, the plurality of point cloud sets comprising a first point cloud set constructed in advance for a charging port area, obtaining a first pose of a mobile charging device through point cloud registration between the plurality of first point clouds and the plurality of point cloud sets, and determining target displacement information according to the first pose and the positional relationship between the first point cloud set and a target point cloud set. The application uses the plurality of point cloud sets to provide a priori information for the search direction of the mobile charging device, so that even if the charging port is invisible at the position of the mobile charging device, the mobile charging device can be guided to the charging port area based on the pose of the mobile charging device and the relative positional relationship between the target point cloud set and the first point cloud set, since the positional relationship between the plurality of point cloud sets (including the first point cloud set of the charging port) is generated in advance.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a charging port positioning method and related device. Background Technology

[0002] With the development and large-scale use of new energy vehicles, mobile charging equipment is increasingly favored by users due to its flexible charging resources and its independence from environmental and parking space constraints. Mobile charging equipment is specifically manifested in the form of mobile charging piles, or in the form of a mobile chassis carrying battery packs. For example, when mobile charging equipment is a mobile charging vehicle, the chassis of the vehicle can carry multiple battery packs. After receiving a user's request, the mobile charging vehicle travels to the user's location and provides charging services for the user's vehicle, completing the charging task.

[0003] Before a mobile charging device can charge a vehicle, it needs to move to the vicinity of the vehicle to locate the charging port area and adjust its own posture (position and direction of the charging arm) to charge the vehicle.

[0004] In existing implementations, mobile charging devices are equipped with monocular cameras. These devices can detect the charging port based on images captured by the monocular camera to locate the charging port area. However, in some scenarios, the parking status of the vehicle to be charged cannot be controlled, and the parking position of the vehicle may cause the charging port area to be outside the field of view of the monocular camera, resulting in positioning failure. In other scenarios, the charging port may be obscured by a charging cover (used to cover the charging port), making it invisible within the field of view of the monocular camera, thus leading to positioning failure. Summary of the Invention

[0005] This application provides a charging port positioning method and apparatus, which can accurately locate the position of the charging port area even when the charging port is not visible at the location of the mobile charging device (or the charging port is blocked by other obstacles).

[0006] In a first aspect, this application provides a charging port positioning method, the method comprising: acquiring a plurality of first point clouds, a plurality of point cloud sets, and the positional relationship between the point cloud sets, wherein the plurality of first point clouds are point clouds collected by sensors on a mobile charging device from a first local area of ​​a target vehicle, each of the point cloud sets is a point cloud set pre-constructed for a local area of ​​the target vehicle, and the plurality of point cloud sets include a first point cloud set, wherein the first point cloud set is a point cloud set pre-constructed for a charging port area;

[0007] Among them, the local area can be a part of the target vehicle with significant visual (structural) features, such as the door area, headlight area, wheel area, charging port area, and license plate area. More specifically, the local area can include the left door area, right door area, left front headlight area, right front headlight area, left rear headlight area, right rear headlight area, left front wheel area, right front wheel area, left rear wheel area, right rear wheel area, charging port area, and license plate area.

[0008] Each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle. The local area can be the surface of the local area, and the point cloud included in each point cloud set can describe the structural features of the surface of the corresponding local area.

[0009] It should be understood that a point cloud set can be understood as a collection of point clouds with certain semantic meaning.

[0010] The so-called positional relationship can be understood as the positional relationship of a point cloud in 3D space. The position of the point cloud can be represented by a feature value that can represent the position of the point cloud in space (for example, it can be represented by the center point of the point cloud). In addition, the positional relationship between point clouds can be represented by the positional relationship between feature values.

[0011] Among them, the plurality of point clouds include a first point cloud, which is a point cloud pre-constructed for the charging port area. The position of the charging port area on the target vehicle can be determined based on the positional relationship between each point cloud.

[0012] Based on the fact that the plurality of first point clouds and the target point cloud set in the plurality of point clouds meet the similarity condition, the first pose of the mobile charging device is obtained by point cloud registration between the plurality of first point clouds and the target point cloud set, wherein the target point cloud set is not the first point cloud set.

[0013] Here, the similarity condition can be understood as follows: when the similarity condition is met, multiple first point clouds and target point cloud sets correspond to the same local area on the target vehicle. The similarity condition can include the similarity of the point cloud distribution and the similarity of the point cloud positions on the target vehicle (which can be determined based on the positional relationship with adjacent areas).

[0014] Based on the first pose and the positional relationship between the first point set and the target point set, target displacement information is determined. The target displacement information is used to guide the mobile charging device to move from the first pose to the charging port area.

[0015] In this embodiment, multiple point cloud sets provide prior information for the search direction of the mobile charging device. Point cloud registration based on multiple first point clouds and target point clouds that meet the similarity conditions in multiple point clouds can calculate the pose of the mobile charging device relative to the target vehicle. Even if the charging port is not visible at the location of the mobile charging device (or the charging port is blocked by other obstacles), since the positional relationship between multiple point cloud sets (including the first point cloud set of the charging port) has been generated in advance, the mobile charging device can be guided to the charging port area based on the pose of the mobile charging device and the relative positional relationship between the target point cloud set and the first point cloud set.

[0016] In one possible implementation, the plurality of point clouds further includes a second point cloud, which is a point cloud pre-constructed for a second local region, the second local region being located between the first local region and the charging port region;

[0017] The step of determining the target displacement information based on the first pose and the positional relationship between the first point set and the target point set includes:

[0018] Based on the first pose and the positional relationship between the second point set and the target point set, the first displacement information in the target displacement information is determined. The first displacement information is used to guide the mobile charging device to move from the first pose to the second local area.

[0019] In one possible implementation, the plurality of point clouds further includes a second point cloud, which is a point cloud pre-constructed for a second local region. The second local region is located between the first local region and the charging port region. Based on the first pose and the positional relationship between the second point cloud and the target point cloud, the first displacement information in the target displacement information can be determined. The first displacement information is used to guide the mobile charging device to move from the first pose to the second local region.

[0020] In one possible implementation, after determining the first displacement information in the target displacement information, multiple second point clouds and pose change information can be obtained. The multiple second point clouds are point clouds collected by the sensor on the second local area of ​​the target vehicle after the mobile charging device moves to the second local area based on the first displacement information. The pose change information is the pose change of the mobile charging device from the first pose to the second local area (e.g., it can be collected by the odometer). The second pose of the sensor is obtained according to the point cloud registration between the second point cloud set and the multiple second point clouds. The positional relationship between the first pose, the second pose, the pose change information, and the second point cloud set and the target point cloud set is used as a closed-loop constraint to perform closed-loop optimization on the second pose to obtain the optimized second pose.

[0021] In this system, each time the mobile charging device moves, it can perform closed-loop optimization of its current pose based on the positional relationships between pre-built point clouds and the pose change information it has collected, thereby improving the accuracy of pose determination.

[0022] In one possible implementation, before the point cloud registration between the plurality of first point clouds and the target point cloud set, the method further includes: obtaining a plurality of third point clouds based on the similarity between the plurality of first point clouds and M point cloud sets in the plurality of point clouds being greater than a threshold, where M is an integer greater than 1, the plurality of third point clouds being point clouds collected by the sensor from a third local region of the target vehicle, the plurality of third point clouds being similar to the third point cloud sets in the plurality of point clouds being greater than a threshold, and the positional relationship between the third local region and the first local region being a first positional relationship; and determining that the plurality of first point clouds and the target point cloud sets in the plurality of point clouds satisfy a similarity condition based on the fact that the positional relationship between the third point cloud sets and the target point cloud sets in the M point cloud sets is the same as the first positional relationship.

[0023] When a unique point cloud set satisfying the similarity condition cannot be matched, the mobile charging device can be guided from the first position to the second position. Although there are multiple point cloud sets with high similarity among the multiple first point clouds, the scope can be narrowed down to the target point cloud set among the multiple point cloud sets based on the relative positional relationship between the local regions corresponding to these multiple point cloud sets and the surrounding local regions. For example, when the first local region is the corner area of ​​the right front of the vehicle, when performing similarity matching on multiple first point clouds, the point cloud sets of the multiple point clouds and the four corner areas on the target vehicle all have a similarity greater than the threshold with the multiple first point clouds. However, the point cloud set of the adjacent area of ​​the first local region in the clockwise direction towards the vehicle body is the point cloud set of the right headlight, and the adjacent area of ​​other corner areas in the clockwise direction towards the vehicle body is not the right headlight (the adjacent area of ​​the left rear corner in the clockwise direction towards the vehicle body is the left door, the adjacent area of ​​the left front corner in the clockwise direction towards the vehicle body is the side surface of the front of the vehicle, or the license plate area of ​​the front of the vehicle, and the adjacent area of ​​the right rear corner towards the vehicle body is the right headlight). The adjacent area in the clockwise direction is the side surface of the rear of the vehicle (or the license plate area of ​​the rear of the vehicle). Therefore, the mobile charging device can be guided to move a certain distance in a certain direction to the second position and collect multiple third point clouds in the third local area. Based on the similarity matching between the multiple third point clouds and the multiple point cloud sets, it is possible to determine which point cloud set is accurate from the multiple point cloud sets. Furthermore, based on the fact that the positional relationship between the third point cloud set and the target point cloud set in the multiple point cloud sets is the same as the first positional relationship, it can be determined that the multiple first point clouds and the target point cloud set in the multiple point cloud sets meet the similarity condition.

[0024] It should be understood that the selection range of a portion of the point cloud set can be narrowed down based on the height information of multiple first point clouds and the ground (generally, the point cloud located at the front of the vehicle will be at a lower height relative to the ground than the point cloud located at the rear of the vehicle). In other words, based on the point cloud distribution, the height above the ground, and the relative relationship between the surrounding area (e.g., the third local area) and the first local area, the point cloud set (target point cloud set) that matches the similarity of multiple first point clouds can be uniquely determined.

[0025] In one possible implementation, the third local area is a door area, a headlight area, a wheel area, or a license plate area.

[0026] In one possible implementation, the first local region is the corner region of the vehicle body, which is formed by two surfaces on the target vehicle that are perpendicular to the ground and mutually perpendicular, and the plurality of first point clouds are the point clouds of the corner region of the vehicle body on a preset height plane.

[0027] In one possible implementation, the first local region is the side surface region of the target vehicle facing forward or backward, and the plurality of first point clouds are point clouds of the side surface region on a preset height plane.

[0028] The preset height can be 20cm, 30cm, 40cm, 50cm, 60cm, etc.

[0029] In one possible implementation, the point cloud range of some areas in the aforementioned regions with significant features is very small (due to the small area of ​​the region itself), and the semantics of the point cloud of the local region (specifically, which part of the target vehicle it is) cannot be directly determined. However, the corner area of ​​the front of the vehicle or the side surface area facing the front or rear of the vehicle, when having significant features, usually appears in the field of view of the sensor due to the large area of ​​the region. Using it as the initial point cloud registration object can improve the success rate of matching.

[0030] In one possible implementation, the plurality of first point clouds are point clouds collected by the sensor from a first local area on the target vehicle when the mobile charging device is in a first position, and the plurality of third point clouds are point clouds collected by the sensor from a third local area on the target vehicle when the mobile charging device is in a second position, wherein the second position is different from the first position; when the mobile charging device determines that the similarity between the first point cloud and multiple sets of first point clouds in the plurality of point clouds is greater than a threshold, that is, when it is impossible to find a unique set of point clouds that meets the similarity condition, the mobile charging device can be guided to move from the first position to the second position.

[0031] The first position and the second position are the positions around the target vehicle, and the second position can be close to the first position (e.g., 20cm, 30cm). This distance can be preset, as long as the sensor of the mobile charging device can collect other local areas (third local areas) besides the first local area at the second position, and is not limited to the area where the second position is located.

[0032] In one possible implementation, the similarity includes: the similarity of the point cloud distribution and / or the similarity of the ground elevation of the point cloud.

[0033] The similarity here can include the similarity of point cloud distribution (e.g., it can be determined based on the step-by-step description or geometric structure description mentioned above), or it can include the similarity of the height of the point cloud above the ground (e.g., the similarity of the height of the point cloud above the ground is high between the point clouds at the front of the vehicle, the similarity of the height of the point cloud above the ground is high between the point clouds at the rear of the vehicle, and the similarity of the height of the point cloud above the ground is low between the point clouds at the front and rear of the vehicle). The similarity of the point cloud distribution can identify regions with similar structural shapes, while the similarity of the height of the point cloud above the ground can distinguish different regions with similar structural shapes on the front and rear of the vehicle, thereby improving the accuracy of pose determination.

[0034] In one possible implementation, the first pose is the relative pose between the sensor and the target vehicle.

[0035] Secondly, this application provides a charging port positioning device, the device comprising:

[0036] The acquisition module is used to acquire multiple first point clouds, multiple point cloud sets, and the positional relationship between each point cloud set. The multiple first point clouds are point clouds collected by sensors on the mobile charging device from a first local area of ​​the target vehicle. Each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle. The multiple point cloud sets include the first point cloud set, which is a point cloud set pre-constructed for the charging port area.

[0037] The pose determination module is used to obtain the first pose of the mobile charging device by registering the multiple first point clouds and the target point cloud set in the multiple point cloud sets according to the similarity condition. The target point cloud set is not the first point cloud set.

[0038] The navigation module is used to determine target displacement information based on the first pose and the positional relationship between the first point set and the target point set. The target displacement information is used to guide the mobile charging device to move from the first pose to the charging port area.

[0039] In the above manner, multiple point clouds provide prior information for the search direction of the mobile charging device. Point cloud registration based on multiple first point clouds and target point clouds that meet the similarity conditions in multiple point clouds can calculate the pose of the mobile charging device relative to the target vehicle. Even if the charging port is not visible at the location of the mobile charging device (or the charging port is blocked by other obstacles), since the positional relationship between multiple point clouds (including the first point cloud of the charging port) has been generated in advance, the mobile charging device can be guided to the charging port area based on the pose of the mobile charging device and the relative positional relationship between the target point cloud and the first point cloud.

[0040] In one possible implementation, the plurality of point clouds further includes a second point cloud, which is a point cloud pre-constructed for a second local region, the second local region being located between the first local region and the charging port region;

[0041] The navigation module is specifically used for:

[0042] Based on the first pose and the positional relationship between the second point set and the target point set, the first displacement information in the target displacement information is determined. The first displacement information is used to guide the mobile charging device to move from the first pose to the second local area.

[0043] In one possible implementation, the acquisition module is further configured to:

[0044] Multiple second point clouds and pose change information are acquired. The multiple second point clouds are point clouds collected by the sensor on the second local area of ​​the target vehicle after the mobile charging device moves to the second local area based on the first displacement information. The pose change information is the pose change of the mobile charging device from the first pose to the second local area.

[0045] The pose determination module is also used for:

[0046] The second pose of the sensor is obtained based on the point cloud registration between the second point cloud set and the plurality of second point clouds;

[0047] Using the first pose, the second pose, the pose change information, and the positional relationship between the second point set and the target point set as closed-loop constraints, the second pose is optimized to obtain the optimized second pose.

[0048] In this system, each time the mobile charging device moves, it can perform closed-loop optimization of its current pose based on the positional relationships between pre-built point clouds and the pose change information it has collected, thereby improving the accuracy of pose determination.

[0049] In one possible implementation, the pose determination module is further configured to: before the point cloud registration between the plurality of first point clouds and the target point cloud set, based on the fact that the similarity between the plurality of first point clouds and M point cloud sets in the plurality of point clouds is greater than a threshold, obtain a plurality of third point clouds, where M is an integer greater than 1, the plurality of third point clouds are point clouds collected by the sensor from a third local region of the target vehicle, the similarity between the plurality of third point clouds and the third point cloud sets in the plurality of point clouds is greater than a threshold, and the positional relationship between the third local region and the first local region is a first positional relationship;

[0050] Based on the fact that the positional relationship between the third point cloud set and the target point cloud set in the M point cloud sets is the same as the first positional relationship, it is determined that the plurality of first point clouds and the target point cloud sets in the plurality of point cloud sets satisfy the similarity condition.

[0051] When a unique point cloud set satisfying the similarity condition cannot be matched, the mobile charging device can be guided from the first position to the second position. Although there are multiple point cloud sets with high similarity among the multiple first point clouds, the scope can be narrowed down to the target point cloud set among the multiple point cloud sets based on the relative positional relationship between the local regions corresponding to these multiple point cloud sets and the surrounding local regions. For example, when the first local region is the corner area of ​​the right front of the vehicle, when performing similarity matching on multiple first point clouds, the point cloud sets of the multiple point clouds and the four corner areas on the target vehicle all have a similarity greater than the threshold with the multiple first point clouds. However, the point cloud set of the adjacent area of ​​the first local region in the clockwise direction towards the vehicle body is the point cloud set of the right headlight, and the adjacent area of ​​other corner areas in the clockwise direction towards the vehicle body is not the right headlight (the adjacent area of ​​the left rear corner in the clockwise direction towards the vehicle body is the left door, the adjacent area of ​​the left front corner in the clockwise direction towards the vehicle body is the side surface of the front of the vehicle, or the license plate area of ​​the front of the vehicle, and the adjacent area of ​​the right rear corner towards the vehicle body is the right headlight). The adjacent area in the clockwise direction is the side surface of the rear of the vehicle (or the license plate area of ​​the rear of the vehicle). Therefore, the mobile charging device can be guided to move a certain distance in a certain direction to the second position and collect multiple third point clouds in the third local area. Based on the similarity matching between the multiple third point clouds and the multiple point cloud sets, it is possible to determine which point cloud set is accurate from the multiple point cloud sets. Furthermore, based on the fact that the positional relationship between the third point cloud set and the target point cloud set in the multiple point cloud sets is the same as the first positional relationship, it can be determined that the multiple first point clouds and the target point cloud set in the multiple point cloud sets meet the similarity condition.

[0052] It should be understood that the selection range of a portion of the point cloud set can be narrowed down based on the height information of multiple first point clouds and the ground (generally, the point cloud located at the front of the vehicle will be at a lower height relative to the ground than the point cloud located at the rear of the vehicle). In other words, based on the point cloud distribution, the height above the ground, and the relative relationship between the surrounding area (e.g., the third local area) and the first local area, the point cloud set (target point cloud set) that matches the similarity of multiple first point clouds can be uniquely determined.

[0053] In one possible implementation, the third local area is a door area, a headlight area, a wheel area, or a license plate area.

[0054] In one possible implementation, the first local region is the corner region of the vehicle body, which is formed by two surfaces on the target vehicle that are perpendicular to the ground and mutually perpendicular, and the plurality of first point clouds are the point clouds of the corner region of the vehicle body on a preset height plane.

[0055] In one possible implementation, the first local region is the side surface region of the target vehicle facing forward or backward, and the plurality of first point clouds are point clouds of the side surface region on a preset height plane.

[0056] In one possible implementation, the point cloud range of some areas in the aforementioned regions with significant features is very small (due to the small area of ​​the region itself), and the semantics of the point cloud of the local region (specifically, which part of the target vehicle it is) cannot be directly determined. However, the corner area of ​​the front of the vehicle or the side surface area facing the front or rear of the vehicle, when having significant features, usually appears in the field of view of the sensor due to the large area of ​​the region. Using it as the initial point cloud registration object can improve the success rate of matching.

[0057] In one possible implementation, the plurality of first point clouds are point clouds collected by the sensor of a first local area on the target vehicle when the mobile charging device is in a first position, and the plurality of third point clouds are point clouds collected by the sensor of a third local area on the target vehicle when the mobile charging device is in a second position, wherein the second position is different from the first position.

[0058] The navigation module is also used for:

[0059] Guide the mobile charging device from the first position to the second position.

[0060] The first position and the second position are the positions around the target vehicle, and the second position can be close to the first position (e.g., 20cm, 30cm). This distance can be preset, as long as the sensor of the mobile charging device can collect other local areas (third local areas) besides the first local area at the second position, and is not limited to the area where the second position is located.

[0061] In one possible implementation, the similarity includes: the similarity of the point cloud distribution and / or the similarity of the ground elevation of the point cloud.

[0062] In one possible implementation, the first pose is the relative pose between the sensor and the target vehicle.

[0063] Thirdly, embodiments of this application provide a computer-readable storage medium, characterized in that it includes computer-readable instructions, which, when executed on a computer device, cause the computer device to perform the first aspect and any of its optional methods.

[0064] Fourthly, embodiments of this application provide a computer program product, characterized in that it includes computer-readable instructions, which, when executed on a computer device, cause the computer device to perform the first aspect and any of its optional methods described above.

[0065] Fifthly, this application provides a chip system including a processor for supporting an execution device or training device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. The chip system may be composed of chips or may include chips and other discrete devices.

[0066] In a sixth aspect, this application provides a mobile charging device, the mobile charging device including a sensor, a driving device, one or more processors and a memory; wherein the memory stores computer-readable instructions; the one or more processors read the computer-readable instructions to control the sensor and the driving device, and execute the methods described in the first aspect and any of its optional methods.

[0067] In a seventh aspect, embodiments of this application provide a server, including: a processor and a memory for storing processor-executable instructions. The processor is configured to execute the instructions to implement the methods described in the first aspect above and any of its alternatives.

[0068] The technical effects brought about by any of the designs in aspects two through seven can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0069] This application provides a charging port positioning method, the method comprising: acquiring multiple first point clouds, multiple point cloud sets, and the positional relationship between the point cloud sets, wherein the multiple first point clouds are point clouds collected by sensors on a mobile charging device from a first local area of ​​a target vehicle, each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle, the multiple point cloud sets include a first point cloud set, the first point cloud set being a point cloud set pre-constructed for a charging port area; based on the similarity condition between the multiple first point clouds and a target point cloud set in the multiple point clouds, obtaining a first pose of the mobile charging device by point cloud registration between the multiple first point clouds and the target point cloud set, wherein the target point cloud set is not a first point cloud set; determining target displacement information based on the first pose and the positional relationship between the first point cloud set and the target point cloud set, the target displacement information being used to guide the mobile charging device to move from the first pose to the charging port area. In the above manner, multiple point clouds provide prior information for the search direction of the mobile charging device. Point cloud registration based on multiple first point clouds and target point clouds that meet the similarity conditions in multiple point clouds can calculate the pose of the mobile charging device relative to the target vehicle. Even if the charging port is not visible at the location of the mobile charging device (or the charging port is blocked by other obstacles), since the positional relationship between multiple point clouds (including the first point cloud of the charging port) has been generated in advance, the mobile charging device can be guided to the charging port area based on the pose of the mobile charging device and the relative positional relationship between the target point cloud and the first point cloud. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0071] Figure 2 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0072] Figure 3 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0073] Figure 4 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0074] Figure 5 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0075] Figure 6 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0076] Figure 7 A flowchart illustrating a charging port positioning method provided in an embodiment of this application;

[0077] Figure 8This application provides a schematic diagram of point cloud model construction as an embodiment.

[0078] Figure 9 This application provides a schematic diagram of point cloud model construction as an embodiment.

[0079] Figure 10 This application provides a schematic diagram of point cloud model construction as an embodiment.

[0080] Figure 11 A partial region illustration provided for an embodiment of this application;

[0081] Figure 12 A partial region illustration provided for an embodiment of this application;

[0082] Figure 13 A partial region illustration provided for an embodiment of this application;

[0083] Figure 14 A partial region illustration provided for an embodiment of this application;

[0084] Figure 15 A partial region illustration provided for an embodiment of this application;

[0085] Figure 16 A partial region illustration provided for an embodiment of this application;

[0086] Figure 17 A partial region illustration provided for an embodiment of this application;

[0087] Figure 18 This is a schematic diagram of a charging port positioning method provided in an embodiment of this application;

[0088] Figure 19 This is a schematic diagram of a charging port positioning method provided in an embodiment of this application;

[0089] Figure 20 This is a schematic diagram of a charging port positioning method provided in an embodiment of this application;

[0090] Figure 21 A schematic diagram of the structure of a charging port positioning device provided in an embodiment of this application;

[0091] Figure 22 A schematic diagram of the structure of a charging port positioning device provided in an embodiment of this application;

[0092] Figure 23 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0093] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0094] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0095] To make this application clearer, we will first give a brief introduction to some of the concepts and processes mentioned in this application.

[0096] 1. Robust:

[0097] The ability of a system to survive abnormal and dangerous conditions. For example, the robustness of computer software refers to its ability to avoid crashing or freezing under conditions of input errors, disk failures, network overload, or intentional attacks. Robustness also refers to the characteristic of a control system to maintain certain performance characteristics under certain parameter perturbations (structural, dimensional).

[0098] 2. Iterative Closest Point (ICP) Algorithm:

[0099] This method is used for registration of two point cloud datasets, involving rotating and translating one of the point cloud datasets so that the intersecting regions of the two datasets completely overlap. The output is a (4x4) rigid body transformation matrix. Essentially, it's an optimal registration method based on least squares. The algorithm repeatedly selects corresponding point pairs and calculates the optimal rigid body transformation until the convergence accuracy requirement for correct registration is met.

[0100] 3. Random Sample Consensus (RANSAC) Algorithm:

[0101] This method iteratively estimates the parameters of a mathematical model from a set of observed data containing outliers. These data points include inliers and outliers. Outliers are not valuable for model estimation; therefore, this method can also be called an outlier detection method. This is a nondeterministic algorithm because it obtains a reasonable result with a certain probability; the probability increases with the number of iterations.

[0102] 4. Kalman Filter: A recursive filtering algorithm (autoregressive filtering algorithm) that can estimate the current state of a dynamic system from a series of incomplete and noisy measurements. The Kalman filter considers the joint distribution of each measurement at different times to generate an estimate of the unknown variable, thus making it more accurate than estimation methods based on a single measurement.

[0103] 5. Two-dimensional (2D) images:

[0104] A two-dimensional image is a planar image that does not contain depth information. Two-dimensional images can include red-green-blue (RGB) images, grayscale images, etc.

[0105] 6. Depth image:

[0106] A depth image, also known as a range image, is an image that uses pixel values ​​representing the distance (or depth) from a depth sensor to various points in space. A depth image directly reflects the geometry of the visible surface of an object in space.

[0107] 7. Point cloud data:

[0108] A point cloud is a collection of points that represent the spatial distribution and surface characteristics of a target object within a given spatial reference frame. In this embodiment, point cloud data is used to characterize the three-dimensional coordinates of each point in the point cloud within a spatial reference coordinate system. This spatial reference coordinate system can be the coordinate system corresponding to the depth sensor.

[0109] 8. Point cloud clusters:

[0110] A point cloud cluster refers to a subset of points in a point cloud that satisfy a predefined partitioning rule, obtained after performing a series of calculations (such as geometric segmentation and clustering) on ​​the point cloud data. The calculation methods can include density-based clustering, kd-tree-based nearest neighbor methods, k-means methods, and deep learning methods, among others.

[0111] In the embodiments of this application, the point cloud data corresponding to a point cloud cluster can be described as a "point cloud set".

[0112] With the development and large-scale use of new energy vehicles, mobile charging equipment is becoming increasingly popular among users due to its flexible charging resources and its independence from environmental and parking space constraints. For example... Figure 1 As shown, mobile charging equipment is specifically manifested in the form of a mobile charging pile, or a mobile chassis carrying a battery pack. For example, when the mobile charging equipment is a mobile charging vehicle, the chassis of the mobile charging vehicle can carry multiple battery packs. After receiving a user's request, the mobile charging vehicle drives to the user's location and provides charging services for the user's vehicle, thus completing the charging task.

[0113] The technical solution of this application can be applied to a mobile charging system, such as... Figure 2 As shown, the system includes a server 11, at least one vehicle system 12, a mobile charging device 13, and a charging pile 14, etc. Each vehicle system 12 may include a user 121, a terminal device 122, and a vehicle 123 to be charged, such as... Figure 3 As shown. Vehicle 123 is the vehicle that user 121 is driving and needs to be charged, and the owner of terminal device 122 is user 121.

[0114] In this scenario, user 121 can be a driver. When the driver discovers that the battery of their vehicle 123 is low, they send a request message to the server via terminal device 122, requesting that the vehicle 123 be charged. Alternatively, user 121 can generate a request on the app on terminal device 122 and then send the request to server 11.

[0115] Terminal device 122 can be a portable device, such as a smart terminal, mobile phone, laptop, tablet computer, personal computer (PC), personal digital assistant (PDA), foldable terminal, wearable device with wireless communication function (such as smartwatch or bracelet), user device (UE), augmented reality (AR) or virtual reality (VR) device, etc. The embodiments of this application do not limit the specific device form of the terminal device.

[0116] Vehicle 123 can be an electric vehicle (EV), which includes a display screen, a vehicle processor and a communication module. In addition, the EV may also include other components or units, which are not limited in this embodiment.

[0117] Optionally, in one possible implementation, the vehicle system 12 may not include the terminal device 122. The functions of the terminal device 122 are implemented by the vehicle's infotainment processor and communication module in the EV. For example, after receiving a user-triggered instruction, the vehicle's infotainment processor sends a request message to the server 11 through the communication module.

[0118] See Figure 2 The server 11 can receive request messages sent by one or more vehicle systems 12, schedule at least one mobile charging device 13 in the system, and dispatch charging tasks to the corresponding mobile charging device, instructing it to drive to the location of the user who sent the request message and charge the requested vehicle. In addition, before scheduling and dispatching charging tasks, the server 11 is also used to obtain road condition information, perform advanced driver assistance system (ADAS) calculations based on the road condition information, and generate scheduling strategies, etc.

[0119] Optionally, the server 11 is a cloud server, which can be a single server or a server cluster consisting of multiple servers.

[0120] The mobile charging device 13 is used to receive instruction signals sent by the server 11, and according to the instruction signals, drive to the location of the vehicle to be charged (which can also be referred to as the target vehicle in this embodiment) and charge it. After the mobile charging device 13 completes the charging task, it returns to the charging pile 14 for recharging. In addition, the mobile charging device 13 is also used to report the current battery pack power status in real time during the charging task, and to report its own location information to the server 11 in real time during the driving process, so that the server 11 can keep track of the status of each mobile charging device in the system in real time, providing a basis for the subsequent dispatch of charging tasks and the scheduling of power resources.

[0121] In addition, each mobile charging device 13 includes a vehicle processor, a communication module, a rechargeable battery or a battery pack. The number of battery packs can be one or more, and each battery pack carries a certain amount of power. In this embodiment, there is no limit to the number of battery packs or the power of each battery pack.

[0122] Optionally, the mobile charging device 13 can be an EV or a gasoline-powered vehicle.

[0123] Before the mobile charging device 13 charges the vehicle to be charged, it needs to move near the vehicle to locate the charging port area on the vehicle and adjust its own posture (position and charging arm direction) to charge the vehicle. In existing implementations, the mobile charging device is equipped with a monocular camera. The mobile charging device can detect the charging port based on the image captured by the monocular camera to locate the charging port area. However, in some scenarios, the parking status of the vehicle to be charged cannot be controlled, and the parking position of the vehicle to be charged makes the charging port area outside the field of view of the monocular camera, resulting in positioning failure. In other scenarios, the charging port is blocked by the charging cover (used to cover the charging port), making the charging port invisible in the field of view of the monocular camera, resulting in positioning failure.

[0124] The charging port positioning method provided in this application embodiment can accurately locate the area where the charging port is located in the above scenario.

[0125] Next, we will introduce the architecture diagram of the mobile charging device 13.

[0126] The embodiments of this application can be applied to, for example... Figure 4 The mobile charging device 13 shown is an example. Figure 4 As shown, the mobile charging device 13 may include: a sensor module 110, a driving device 120, a charging device 140, and a main control platform 130.

[0127] The sensor module 110 may include one or more cameras 111 (which may also be referred to as sensors in this embodiment), such as ordinary optical cameras, infrared cameras, structured light sensors, or time-of-flight (ToF) sensors. For example, the sensor module 110 may include an ordinary RGB camera or a red-yellow-yellow-blue (RYYB) camera. The camera device module may also include multiple cameras or sensors forming an RGB-D depth camera solution. For example, the RGB-D depth camera solution may include two RGB cameras forming a binocular solution, one RGB camera and one structured light sensor forming a structured light solution, or one RGB camera and one ToF sensor forming a ToF solution; this embodiment does not specifically limit the specifics. Furthermore, the camera 111 may be a fixed-focus camera or a zoom camera, for example, possessing phase detection autofocus, laser autofocus, or other similar capabilities.

[0128] It should be understood that the camera 111 can be mounted on a motion unit, which carries the camera 111 and drives it to rotate. In one embodiment, the motion unit can drive the camera 111 to rotate with two degrees of freedom. If the Z-axis points in the direction directly in front of the camera 11, then the two degrees of freedom rotation can include the rotation of the camera 111 about the x-axis and the rotation of the camera 111 about the y-axis. The motion unit can drive the camera 111 to rotate by the rotation of a servo motor or a servo motor. For example, when the drive device is used to drive the camera 111 to rotate with two degrees of freedom, the motion unit can include two drive mechanisms, driver 1 and driver 2, such as two servo motors or two servo motors. One servo motor 1 or servo motor 1 is used to control the rotation of the camera 111 about the x-axis, and the other servo motor 2 or servo motor 2 is used to control the rotation of the camera 111 about the y-axis. In other embodiments, the motion unit can drive the camera 111 to generate three degrees of freedom rotation, that is, increase the rotation of the camera 111 about the z-axis. Accordingly, the motion unit may also include three drive mechanisms driver 1, driver 2 and driver 3, such as three servo motors or three servo motors. One servo motor 1 or servo motor 1 is used to control the rotation of the camera 111 about the x-axis, another servo motor 2 or servo motor 2 is used to control the rotation of the camera 111 about the y-axis, and another servo motor 3 or servo motor 3 is used to control the rotation of the camera 111 about the z-axis.

[0129] The sensor module 110 may also include a motion sensor 112, which may be an odometer, accelerometer, speedometer, inertial measurement unit, etc., used to collect mileage information such as distance, trajectory, and speed of the mobile charging device 13 during its operation.

[0130] The drive unit 120 may include components that provide powered motion to the mobile charging device 13. In one embodiment, the drive unit 120 may include an engine, an energy source, a transmission, and wheels / tires. The engine may be an internal combustion engine, an electric motor, an air compressor engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compressor engine. The engine converts the energy source into mechanical energy.

[0131] Examples of energy sources include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. The energy source can also power other systems within the mobile charging device 13.

[0132] A transmission system can transmit mechanical power from an engine to the wheels. The transmission system may include a differential and a drive shaft. In one embodiment, the transmission system may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels.

[0133] The main control platform 130 is the data processing and control center of the device. The main control platform 130 establishes communication connections with the charging device 140, the sensor module 110, and the drive device 120. For example, it can receive image data collected by the sensor module 110, process the image data, and send movement commands to the drive device 120. The charging device 140 may include a charging head 141 and a robotic arm 142. In some embodiments, the main control platform 130 may be an embedded computer platform, including but not limited to computer chips and software systems designed based on x86, ARM, RISC-V, or MIPS instruction sets.

[0134] In one embodiment, the aforementioned computer chip may include, for example, a processor 131 and a memory 132. The processor 131 may include, for example, a central processing unit (CPU), a system-on-a-chip (SoC), an application processor (AP), a microcontroller, a neural-network processing unit (NPU), and / or a graphics processing unit (GPU). The memory 132 may include, for example, non-volatile memory and volatile memory. Non-volatile memory may include flash memory, such as NAND flash, solid-state drives, etc., and volatile memory may include synchronous dynamic random-access memory (SDRAM), etc.

[0135] In one embodiment, the software system may include an operating system and program instructions 133 running on the operating system. When the processor executes the program instructions, it causes... Figure 3 or Figure 4 The device shown performs each step of the charging port positioning method provided in the embodiments of this application.

[0136] In some embodiments, memory 132 may contain program instructions 133 (e.g., program logic) that can be executed by processor 131 to perform various functions of the mobile charging device 13, including those described above. Memory 132 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the drive unit 120, sensor module 110, control system, and peripheral devices.

[0137] In addition to program instructions 133, memory 132 may also store data such as road maps, route information, the location, direction, speed, and other data of the autonomous driving device, as well as other information. This information can be used by the mobile charging device 13 during operation in autonomous, semi-autonomous, and / or manual modes.

[0138] The wireless communication system 150 can communicate wirelessly with one or more devices (e.g., server 11) directly or via a communication network. For example, the wireless communication system 150 can use 3G cellular communication, such as code division multiple access (CDMA), EVDO, Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or 4G cellular communication, such as long term evolution (LTE), or 5G cellular communication. The wireless communication system 150 can communicate using WiFi and a wireless local area network (WLAN). In some embodiments, the wireless communication system 150 can communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various autonomous driving device communication systems, may also be used. For example, the wireless communication system 150 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between autonomous driving devices and / or roadside stations.

[0139] Optionally, one or more of these components may be installed separately from or associated with the mobile charging device 13. For example, the memory 132 may exist partially or completely separately from the mobile charging device 13. The components may be communicatively coupled together in a wired and / or wireless manner.

[0140] Optionally, the components described above are merely examples. In actual applications, components in each of the above modules may be added or removed as needed. Figure 4 This should not be construed as a limitation on the embodiments of this application.

[0141] In this embodiment of the application, the executing entity of the charging port positioning method can be a mobile charging device 13 or a cloud-based server 11, as shown in the reference. Figure 5 , Figure 5 The system shown can have a mobile charging device 13 and a server 11, which are connected in communication (e.g., via a wireless communication system 150). The mobile charging device 13 can transmit data collected by the sensor to the server 11, which then uses the data collected by the sensor to implement the charging port positioning method in this embodiment and transmits the positioning result back to the mobile charging device 13.

[0142] The following is a schematic diagram of the architecture of server 11.

[0143] This application also provides a server; please refer to [link / reference]. Figure 6 , Figure 6 This is a schematic diagram of a server structure provided in an embodiment of this application. Specifically, server 600 is implemented by one or more servers. Server 600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 66 (e.g., one or more processors) and memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) for storing application programs 642 or data 644. The memory 632 and storage media 630 can be temporary or persistent storage. The program stored in storage media 630 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 66 may be configured to communicate with storage media 630 and execute the series of instruction operations in storage media 630 on server 600.

[0144] Server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input / output interfaces 658; or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0145] In this embodiment, the central processing unit 66 is used to execute the charging port positioning method described in the above embodiments.

[0146] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0147] Reference Figure 7 , Figure 7 This is a schematic diagram of an embodiment of a charging port positioning method provided in this application. The charging port positioning method provided in this application can be applied to mobile charging devices or servers, wherein the mobile charging device can be a mobile charging vehicle, a mobile charging robot, or other products. Figure 7 As shown, the charging port positioning method provided in this application embodiment may include:

[0148] 701. Obtain multiple first point clouds, multiple point cloud sets, and the positional relationship between each point cloud set. The multiple first point clouds are point clouds collected by sensors on the mobile charging device from a first local area of ​​the target vehicle. Each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle. The multiple point cloud sets include the first point cloud set, which is a point cloud set pre-constructed for the charging port area.

[0149] The entity executing step 701 can be a mobile charging device or a cloud-based server; there are no specific limitations on this.

[0150] In one possible implementation, the mobile charging device can acquire multiple first point clouds and transmit them to a server. The server can store multiple point cloud sets and the positional relationships between each point cloud set. The multiple first point clouds are point clouds collected by sensors on the mobile charging device from a first local area of ​​the target vehicle. Each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle. The multiple point cloud sets include the first point cloud set, which is a point cloud set pre-constructed for the charging port area.

[0151] In one possible implementation, the mobile charging device can acquire multiple first point clouds, multiple point cloud sets, and the positional relationships between the point cloud sets.

[0152] In this embodiment of the application, when the mobile charging device locates the charging port of the vehicle to be charged (also referred to as the target vehicle in this embodiment of the application), it needs to determine the relative pose between itself and the target vehicle, as well as the position of the charging port on the target vehicle, and then perform positioning and navigation based on the relative pose and the pose of the charging port.

[0153] In one possible implementation, the mobile charging device can collect point cloud data of the target vehicle using its own deployed sensors, and determine the relative pose between the mobile charging device and the target vehicle by point cloud registration between pre-built vehicle point cloud models.

[0154] In this application, the point cloud, also referred to as point cloud data, is a set of points that represent the spatial distribution and surface characteristics of a target under the same spatial reference system. After obtaining the spatial coordinates of each sampling point on the object's surface, a set of points is obtained, called a point cloud. In this application, point cloud data is used to characterize the three-dimensional coordinates of each point in the point cloud data under a reference coordinate system. Furthermore, in some embodiments of this application, point cloud data can also be fused with pixels from an RGB image. Therefore, in some embodiments of this application, point cloud data can also be used to characterize the pixel value of each point in the point cloud data and the three-dimensional coordinates of each point under a reference coordinate system.

[0155] Next, we will introduce the pre-built vehicle point cloud model:

[0156] In one possible implementation, the vehicle point cloud model can be constructed in advance from data collected by sensors on the vehicle. It should be understood that the structure of different vehicle models may be different. For example, the size, surface characteristics of each local area, and the position of each local area on the vehicle (e.g., the position of the charging port on the vehicle) may all be different. Therefore, separate vehicle point cloud models can be constructed for different vehicle models. For example, when a mobile charging device acquires the point cloud model of a target vehicle, it can select a model that matches the target vehicle from a model library.

[0157] In one possible implementation, the pre-built vehicle point cloud model may include multiple point cloud sets and the positional relationships between the point cloud sets. Each point cloud set is a point cloud set pre-built for a local region of the target vehicle, and in order to characterize the overall features of the target vehicle, the vehicle point cloud model also needs to include the positional relationships between the point cloud sets.

[0158] In one possible implementation, each point cloud set is a point cloud set pre-constructed for a local region of the target vehicle, where the local region can be the surface of the local region, and the point cloud included in each point cloud set can describe the structural features of the surface of the corresponding local region.

[0159] In one possible implementation, the so-called positional relationship can be understood as the positional relationship of a point cloud in 3D space. The position of the point cloud can be represented by a feature value that can represent the position of the point cloud in space (for example, by the center point of the point cloud). Furthermore, the positional relationship between point clouds can be represented by the positional relationship between feature values.

[0160] In one possible implementation, the local area can be a part of the target vehicle with significant visual (structural) features, such as the door area, headlight area, wheel area, charging port area, and license plate area. More specifically, the local area can include the left door area, right door area, left front headlight area, right front headlight area, left rear headlight area, right rear headlight area, left front wheel area, right front wheel area, left rear wheel area, right rear wheel area, charging port area, and license plate area.

[0161] In one possible implementation, the plurality of point cloud sets includes a first point cloud set, which is a point cloud set pre-constructed for the charging port area. The point cloud set included in the first point cloud set can express the distribution characteristics and surface properties of the charging port area in three-dimensional space. In addition, since the vehicle point cloud model includes the positional relationship between each point cloud set, the vehicle point cloud model can include the positional relationship between the first point cloud set and other point cloud sets, that is, the position of the charging port area on the target vehicle.

[0162] In one possible implementation, the plurality of point clouds includes a first point cloud, which is a point cloud pre-constructed for the charging port area. The location of the charging port area on the target vehicle can be determined based on the positional relationship between the various point clouds.

[0163] The following is a schematic diagram of the specific data representation of a vehicle point cloud model:

[0164] In one possible implementation, the aforementioned multiple point cloud sets can constitute a semantic topology tree. Each child node (or topology node) on the semantic topology tree corresponds to one or more local regions. Each child node can include multiple point clouds (or be described as including at least one point cloud set), corresponding semantics (semantics can be understood as the name of the local region), and feature descriptors (feature descriptors can be understood as the distribution characteristics of the point cloud, and the feature descriptors can be possessed by some or all of the child nodes).

[0165] A semantic topology tree can include child nodes at multiple levels. Higher-level child nodes have a larger granularity (including more local regions) than lower-level child nodes. For example, the left half of the car body is child node 1, the right half of the car body is child node 2, the left door is child node 3, and the right door is child node 4. Child nodes 1 and 2 are at a higher level than child nodes 3 and 4.

[0166] For example, a semantic topology tree can be constructed as follows: the target vehicle is divided into a left half and a right half, serving as the first-level nodes. Based on the left half, a second-level node is created, consisting of the left front, left door, and left rear. This layer-by-layer division defines the model as a collection of multiple nodes. Furthermore, the bottom-level child nodes of the topology nodes can correspond to multiple parts of the vehicle body. To express the positional relationships between these child nodes, a 3D connection graph can be generated based on each 3D part (e.g., headlights, doors, charging ports, wheels, license plates). 3D parts with connections on the 3D connection graph can be defined as those visible within the same field of view. Optionally, the 3D coordinates of each child node are the center point of the corresponding point cluster. Figure 8 As shown, Figure 8 This is a schematic diagram of a 3D connection graph, which can be used for dynamic path search.

[0167] For the point cloud of each topological node, the data format can be 3D coordinates, such as [x, y, z] (excluding color information) to represent the surface point set of the region. The center point can be obtained by calculating the mean of the 3D coordinates.

[0168] In one possible implementation, the feature descriptors of each topological node may include, but are not limited to, both geometric structure descriptions and distribution descriptions. The geometric structure description is the 2D contour data (e.g., a preset height) extracted from the point cloud of the topological region at a certain height H perpendicular to the ground. Figure 9 As shown), the descriptor is defined as {d, θ} relative to the vehicle coordinate system, where d represents the distance from point O to the vehicle center point C, and θ represents the angle of counterclockwise rotation relative to the vehicle coordinate Z-axis. Furthermore, if more than two nodes are observed, they can be defined pairwise, with center point O defined as the midpoint of the line connecting the center points of two topological nodes, and projected onto a 2D plane of height H, as shown. Figure 10 As shown. The description method can be defined as {d, θ}, where d is the distance from 0C, and θ can be defined by referring to θ in the 2D contour data perpendicular to the ground at a certain height H (preset height) mentioned above.

[0169] In one possible implementation, the distribution descriptor can represent the distribution state of the node's own point set, as follows: Based on the query point Pq, points in the r neighborhood can be found and the (relative angle) between point pairs can be calculated. The query point represents all points in each topological node that are within 90% (or other proportions close to 1) of the distance from the center point.

[0170] The following sections introduce several methods for collecting first-point clouds:

[0171] When a mobile charging device starts a charging task, it can acquire multiple frames of point cloud data observed at the moment, generate a local point cloud map, remove ground points using methods such as RANSAC, and segment the point cloud data (multiple first point clouds) containing the target vehicle in the local map using point cloud clustering methods such as DBSCAN.

[0172] Clustering algorithms belong to unsupervised learning, where no specific classification category is given; the categories are determined through similarity. Nearest neighbor clustering, on the other hand, is a clustering algorithm based on a distance threshold. For example, suppose there are N point cloud data points {X1, X2, ..., Xn} to be classified, and the goal is to classify them into clusters centered at {Z1, Z2, ...} according to a distance threshold T. The process of the nearest neighbor clustering algorithm is as follows: First, arbitrarily select a sample Xi as the initial value for the first cluster center, such as letting Z1 = X1. Second, calculate the Euclidean distance D21 = ||X2 - Z1|| from sample X2 to Z1. If D21 > T, then define a new cluster center Z2 = X2; otherwise, X2 belongs to the cluster centered at Z1. The third step, assuming there are already cluster centers {Z1, Z2}, calculate D31 = ||X3 - Z1|| and D32 = ||X3 - Z2||. If D31 > T and D32 > T, then establish a third cluster center Z3 = X3; otherwise, X3 belongs to the cluster closest to Z1 and Z2 (i.e., the nearest neighbor cluster center). Note that this method of selecting the cluster center closest to itself is called the nearest neighbor clustering algorithm. The fourth step, and so on, continues until all N samples have been classified.

[0173] Among them, multiple first point clouds can be point clouds collected by sensors on mobile charging devices at a certain moment based on a certain pose from the first local area on the target vehicle.

[0174] It should be understood that multiple first point clouds can be point clouds that meet certain distribution characteristics selected from the point cloud data of the target vehicle.

[0175] For ease of understanding, Figure 11 For example, the monitoring range of the camera deployed on this mobile charging device at the current moment is as follows: Figure 11 As shown in the cone, within this monitoring range, there is a corner at the front of the vehicle. The mobile charging device will then acquire the point cloud (i.e., multiple first point clouds) corresponding to this local area.

[0176] In one possible implementation, the mobile charging device can collect information from the vehicle being charged based on its own deployed sensors to obtain multiple first point clouds, wherein the sensors can be depth sensors used to acquire depth images. Depth sensors can be, for example, but not limited to, depth cameras, time-of-flight (TOF) cameras, or lidar, photographic scanners, or light detection and ranging (LiDAR) sensors.

[0177] It should be noted that, in some embodiments of this application, at the current moment, the point cloud corresponding to each local area within the current location monitoring range acquired by the mobile charging device is obtained based on the raw data collected by the relevant sensors deployed on the mobile charging device. Specifically, based on the collected raw data, the mobile charging device processes the raw data to obtain the point cloud data corresponding to each local area within the monitoring range. Taking multiple first point clouds as an example, the mobile charging device can acquire multiple first point clouds in, but is not limited to, the following ways:

[0178] A. Multiple first point clouds are obtained based on the acquired RGB and depth images.

[0179] In this embodiment, the mobile charging device can acquire RGB images and depth images through sensors deployed on the device. RGB images and depth images refer to the RGB images and depth images formed by the surrounding environment that the sensors can perceive at the current time and current location. For example, RGB images and corresponding depth images can be acquired by a depth camera deployed on the mobile charging device, or depth images can be acquired by a depth sensor deployed on the device, and the corresponding RGB images can be acquired by a camera module (e.g., a webcam) deployed on the device.

[0180] After the sensor acquires the RGB and depth images of the mobile charging device at the current moment, it first performs instance segmentation on the RGB image. For example, algorithms such as SOLOv2 and BlendMask can be used to perform instance segmentation on the RGB image, and the instance segmentation result is output. The instance segmentation result includes the classification category (i.e., label, such as chair, person, etc.) to which each local region in the RGB image belongs, the confidence score (not greater than 1) of belonging to that classification category, and the segmentation mask. The instance segmentation result can be defined as Mi. Then, the data of the local region belonging to the target vehicle can be selected, and the RGB image that has been segmented and whose pixel values ​​have been redefined (which can be called the processed RGB image) is superimposed on the depth image. Each local region in the processed RGB image then has depth information. Specifically, since the area occupied by the mask corresponding to each local region in the processed RGB image is defined as 1 pixel and the rest of the area is defined as 0 pixels, when the processed RGB image is multiplied by the depth image, since the area with a pixel value of 0 is still 0 after multiplication, the result of multiplication only retains the area corresponding to each local region in the RGB image and the depth information corresponding to each local region. Then, the point cloud data of each local area is recovered using camera intrinsic parameters.

[0181] It should be noted that in some embodiments of this application, the instance segmentation results and the depth image can be filtered first, such as reducing the outliers of the instance segmentation results through morphological filtering and eliminating the holes in the depth image through depth smoothing. In short, the purpose of filtering is to remove noise.

[0182] B. Multiple first point clouds are obtained based on the acquired RGB images and original laser point cloud data.

[0183] In this embodiment, the mobile charging device can acquire RGB images and raw laser point cloud data through sensors deployed on the device. For example, the mobile charging device can be equipped with sensors such as LiDAR and ordinary cameras. The camera can acquire RGB images, and the LiDAR can acquire raw laser point cloud data. In this case, it is not necessary to acquire depth images. The RGB images can be segmented into instances according to a similar process, and the segmentation results can be mapped to the raw laser point cloud data acquired at the same time to obtain multiple first point clouds.

[0184] 702. Based on the fact that the plurality of first point clouds and the target point cloud set in the plurality of point clouds meet the similarity condition, the first pose of the mobile charging device is obtained by point cloud registration between the plurality of first point clouds and the target point cloud set, wherein the target point cloud set is not the first point cloud set.

[0185] The entity executing step 702 can be a mobile charging device or a cloud-based server; there are no specific limitations on this.

[0186] In one possible implementation, the mobile charging device can obtain the first pose of the mobile charging device by registering the multiple first point clouds and the target point cloud set in the multiple point cloud sets to meet the similarity condition, where the target point cloud set is not a first point cloud set.

[0187] In one possible implementation, the server can obtain the first pose of the mobile charging device by registering the multiple first point clouds and the target point cloud set in the multiple point cloud sets to meet the similarity condition, where the target point cloud set is not a first point cloud set.

[0188] In this embodiment of the application, in order to obtain the current relative pose of the sensor relative to the target vehicle, similarity matching can be performed between multiple first point clouds and the multiple point cloud sets. The mobile charging device obtains the first pose of the mobile charging device by registering the multiple first point clouds with the target point cloud set based on the similarity condition between the multiple first point clouds and the target point cloud set. The first pose can be the relative pose of the sensor relative to the target vehicle when it acquires multiple first point clouds. Since the sensor is fixed on the mobile charging device, the first pose of the mobile charging device can be determined based on the pose of the sensor.

[0189] Here, the similarity condition can be understood as follows: when the similarity condition is met, multiple first point clouds and target point cloud sets correspond to the same local area on the target vehicle. The similarity condition can include the similarity of the point cloud distribution and the similarity of the point cloud positions on the target vehicle (which can be determined based on the positional relationship with adjacent areas).

[0190] In one possible implementation, point cloud registration can be performed based on regions with significant features on the target vehicle. Point cloud registration based on these regions can quickly determine the current pose of the sensor. For example, regions with significant features can be headlight areas, door areas, charging port areas, wheel areas, license plate areas, corner areas at the front or rear of the vehicle, side surface areas facing the front or rear of the vehicle, etc.

[0191] For example, if multiple first point clouds are point clouds of the vehicle headlight region, a point cloud set (target point cloud set) with high similarity to multiple first point clouds can be selected from multiple point cloud sets, and the first pose of the mobile charging device can be obtained based on the point cloud registration between multiple first point clouds and the target point cloud set.

[0192] It should be understood that the similarity here can include the similarity of point cloud distribution (e.g., it can be determined based on the step-by-step description or geometric structure description mentioned above), and it can also include the similarity of the height of the point cloud above the ground (e.g., the similarity of the height of the point cloud above the ground is high between the point clouds at the front of the vehicle, the similarity of the height of the point cloud above the ground is high between the point clouds at the rear of the vehicle, and the similarity of the height of the point cloud above the ground is low between the point clouds at the front and rear of the vehicle). The similarity of the point cloud distribution can identify regions with similar structural shapes, while the similarity of the height of the point cloud above the ground can distinguish different regions with similar structural shapes on the front and rear of the vehicle, thereby improving the accuracy of pose determination.

[0193] In one possible implementation, the point cloud range of some areas in the aforementioned regions with significant features is very small (due to the small area of ​​the region itself), and the semantics of the point cloud of the local region (specifically, which part of the target vehicle it is) cannot be directly determined. However, the corner areas of the front or rear of the vehicle, or the side surface areas facing the front or rear of the vehicle, have significant features and are usually in the field of view of the sensor due to their large area. Using them as the initial point cloud registration objects can improve the success rate of matching.

[0194] In one possible implementation, the first local region is a corner region of the vehicle body, the corner region being formed by two mutually perpendicular surfaces perpendicular to the ground on the target vehicle, and the plurality of first point clouds being point clouds of the corner region of the vehicle body on a preset height plane. The preset height can be 20cm, 30cm, 40cm, 50cm, 60cm, etc.

[0195] In one possible implementation, the first local region is the corner region of the vehicle body, which is formed by two surfaces on the target vehicle that are perpendicular to the ground and mutually perpendicular, and the plurality of first point clouds are the point clouds of the corner region of the vehicle body on a preset height plane.

[0196] Reference Figures 12 to 15 , Figures 12 to 15 A schematic diagram of the four corner areas on the target vehicle is shown.

[0197] In one possible implementation, the first local region is the side surface region of the target vehicle facing forward or backward, and the plurality of first point clouds are point clouds of the side surface region on a preset height plane.

[0198] Reference Figure 16 and Figure 17 , Figure 16 and Figure 17 A schematic diagram of a side surface area on the target vehicle facing forward or backward is shown.

[0199] In one possible implementation, since the target vehicle is a symmetrical structure, the similarity between the left front corner region, left rear corner region, right front corner region, and right rear corner region on the target vehicle is very high. For example, if multiple first point clouds are left front corner regions, when calculating the similarity with multiple point cloud sets, it can be found that the similarity between multiple first point clouds and multiple first point cloud sets is greater than the threshold. In this case, it is necessary to determine from the multiple first point cloud sets which is the point cloud set that satisfies the similarity condition with the multiple first point clouds.

[0200] In one possible implementation, multiple third point clouds can be obtained based on the fact that the similarity between the multiple first point clouds and M point cloud sets in the multiple point cloud set is greater than a threshold, where M is an integer greater than 1. The multiple third point clouds are point clouds collected by the sensor from the third local region of the target vehicle. The similarity between the multiple third point clouds and the third point cloud sets in the multiple point cloud set is greater than a threshold. The positional relationship between the third local region and the first local region is a first positional relationship.

[0201] In one possible implementation, the plurality of first point clouds are point clouds collected by the sensor from a first local area on the target vehicle when the mobile charging device is in a first position, and the plurality of third point clouds are point clouds collected by the sensor from a third local area on the target vehicle when the mobile charging device is in a second position, wherein the second position is different from the first position; when the mobile charging device determines that the similarity between the first point cloud and multiple sets of first point clouds in the plurality of point clouds is greater than a threshold, that is, when it is impossible to find a unique set of point clouds that meets the similarity condition, the mobile charging device can be guided to move from the first position to the second position.

[0202] Wherein, the first position and the second position are the positions around the target vehicle, and the second position can be relatively close to the first position (e.g., 20cm, 30cm). This distance can be preset, as long as it is ensured that the sensor of the mobile charging device can collect other local areas (third local areas) besides the first local area at the second position, and is not limited to the area where the second position is located.

[0203] When a unique point cloud set satisfying the similarity condition cannot be matched, the mobile charging device can be guided from the first position to the second position. Although there are multiple point cloud sets with high similarity among the multiple first point clouds, the scope can be narrowed down to the target point cloud set among the multiple point cloud sets based on the relative positional relationship between the local regions corresponding to these multiple point cloud sets and the surrounding local regions. For example, when the first local region is the corner area of ​​the right front of the vehicle, when performing similarity matching on multiple first point clouds, the point cloud sets of the multiple point clouds and the four corner areas on the target vehicle all have a similarity greater than the threshold with the multiple first point clouds. However, the point cloud set of the adjacent area of ​​the first local region in the clockwise direction towards the vehicle body is the point cloud set of the right headlight, and the adjacent area of ​​other corner areas in the clockwise direction towards the vehicle body is not the right headlight (the adjacent area of ​​the left rear corner in the clockwise direction towards the vehicle body is the left door, the adjacent area of ​​the left front corner in the clockwise direction towards the vehicle body is the side surface of the front of the vehicle, or the license plate area of ​​the front of the vehicle, and the adjacent area of ​​the right rear corner towards the vehicle body is the right headlight). The adjacent area in the clockwise direction is the side surface of the rear of the vehicle (or the license plate area of ​​the rear of the vehicle). Therefore, the mobile charging device can be guided to move a certain distance in a certain direction to the second position and collect multiple third point clouds in the third local area. Based on the similarity matching between the multiple third point clouds and the multiple point cloud sets, it is possible to determine which point cloud set is accurate from the multiple point cloud sets. Furthermore, based on the fact that the positional relationship between the third point cloud set and the target point cloud set in the multiple point cloud sets is the same as the first positional relationship, it can be determined that the multiple first point clouds and the target point cloud set in the multiple point cloud sets meet the similarity condition.

[0204] It should be understood that the selection range of a portion of the point cloud set can be narrowed down based on the height information of multiple first point clouds and the ground (generally, the point cloud located at the front of the vehicle will be at a lower height relative to the ground than the point cloud located at the rear of the vehicle). In other words, based on the point cloud distribution, the height above the ground, and the relative relationship between the surrounding area (e.g., the third local area) and the first local area, the point cloud set (target point cloud set) that matches the similarity of multiple first point clouds can be uniquely determined.

[0205] In one possible implementation, the third local area is a door area, a headlight area, a wheel area, or a license plate area.

[0206] In this embodiment of the application, after determining the target point cloud set, the first pose of the mobile charging device can be obtained by point cloud registration between the plurality of first point clouds and the target point cloud set.

[0207] The first pose can be the relative pose between the sensor and the target vehicle.

[0208] Point cloud registration, also known as 3D point cloud registration, is a key research problem in computer vision and has important applications in various engineering fields, such as reverse engineering, simultaneous localization and mapping (SLAM), image processing, and pattern recognition. The purpose of point cloud registration is to solve for the transformation matrix of point clouds in different poses under the same coordinate system. This matrix is ​​then used to achieve accurate registration of multi-view scanning point clouds, ultimately obtaining a complete 3D digital model and scene. The Intermediate Point Collation (ICP) algorithm is generally applied to the registration of two point cloud datasets. The basic principle of the ICP algorithm is: in the target point cloud P and the source point cloud Q, respectively, according to certain constraints, find the nearest neighbor point (pi, qi), and then calculate the optimal matching parameters R and t to minimize the error function. pi is a point in the target point cloud P, qi is the nearest point in the source point cloud Q corresponding to pi, R is the rotation matrix, and t is the translation vector.

[0209] For example, assuming we have two 3D point cloud datasets, X1 and X2, the point cloud registration process based on the ICP algorithm is as follows: First, calculate the nearest point in X1 for each point in X2, obtaining corresponding point pairs. Second, find the rigid body transformation matrix that minimizes the average distance between these corresponding point pairs, and calculate the translation vector and rotation angle matrix. Third, apply the translation vector and rotation matrix obtained in the previous step to X2 to obtain a new set of transformed points. Fourth, if the average distance between the new set of transformed points and the reference set is less than a given threshold, stop the iterative calculation; otherwise, use the new set of transformed points as the new X2 and continue iterating until the objective function requirement is met.

[0210] For more details on point cloud registration, please refer to the existing technology descriptions; they will not be repeated here.

[0211] 703. Based on the first pose and the positional relationship between the first point set and the target point set, determine the target displacement information, which is used to guide the mobile charging device to move from the first pose to the charging port area.

[0212] The entity executing step 703 can be a mobile charging device or a cloud-based server; there are no specific limitations on this.

[0213] In one possible implementation, the mobile charging device can determine target displacement information based on the first pose and the positional relationship between the first point set and the target point set. The target displacement information is used to guide the mobile charging device to move from the first pose to the charging port area.

[0214] In one possible implementation, the cloud-side server can determine target displacement information based on the first pose and the positional relationship between the first point set and the target point set. The target displacement information is used to guide the mobile charging device to move from the first pose to the charging port area.

[0215] In this embodiment, after obtaining the first pose, it is equivalent to knowing the relative positional relationship between the mobile charging device and the target vehicle. Since the relative positional relationship between the point clouds in the multiple point clouds can characterize the position of the charging port area on the target vehicle, the relative positional relationship between the mobile charging device and the charging port area can be determined based on the first pose and the position of the charging port area on the target vehicle, and the mobile charging device can be guided to move in the direction of the charging port area.

[0216] In one possible implementation, during movement, the pose can be optimized based on point cloud registration with the local regions traversed along the path.

[0217] Specifically, taking the movement of a mobile charging device along a 3D connection map as an example, the mobile charging device can move to the next topology node based on the position mode. After obtaining the initial target point pose, it moves forward along an ordered topology sequence through the 3D connection map. The forward movement mode is the position mode, which only requires the wheel odometer data.

[0218] In one possible implementation, the plurality of point clouds further includes a second point cloud, which is a point cloud pre-constructed for a second local region. The second local region is located between the first local region and the charging port region. Based on the first pose and the positional relationship between the second point cloud and the target point cloud, the first displacement information in the target displacement information can be determined. The first displacement information is used to guide the mobile charging device to move from the first pose to the second local region.

[0219] In one possible implementation, after determining the first displacement information in the target displacement information, multiple second point clouds and pose change information can be obtained. The multiple second point clouds are point clouds collected by the sensor on the second local area of ​​the target vehicle after the mobile charging device moves to the second local area based on the first displacement information. The pose change information is the pose change of the mobile charging device from the first pose to the second local area. The second pose of the sensor is obtained according to the point cloud registration between the second point cloud set and the multiple second point clouds. The positional relationship between the first pose, the second pose, the pose change information, and the second point cloud set and the target point cloud set is used as a closed-loop constraint to perform closed-loop optimization on the second pose to obtain the optimized second pose.

[0220] This involves establishing closed-loop constraints for topological nodes and optimizing their poses. When the mobile charging device moves to the next topological node, the relative pose of the current observation distance model can be estimated using structural registration and ICP algorithms. The relative poses of the two topological nodes can then be obtained from a wheel-based odometer. The relative pose constraint is defined as an edge, corresponding to... Figure 19 A triangular region is considered. Least squares optimization can be performed on multiple triangle edges to minimize the cumulative worst-case scenario. Since the model is a unified whole, each optimization adjusts the global pose of the model, aiming to minimize the sum of squared errors across all edges. For example, the least squares optimization (Bundle Adjustment) constraint can be illustrated as follows:

[0221]

[0222]

[0223] Reference Figure 19 Each time the device traverses a local area, the aforementioned closed-loop optimization is performed to optimize the pose of the mobile charging device. Based on the optimized pose, the next displacement information is determined until the mobile charging device moves from the first pose to the charging port area. Through closed-loop optimization, the pose accuracy of the mobile charging device can be improved, which in turn improves the positioning accuracy of the charging port area.

[0224] The following describes the charging port positioning method provided in this application embodiment using a specific example:

[0225] Reference Figure 20When the mobile charging device begins its charging task, it moves to the vicinity of the target vehicle, acquires multi-frame point cloud data, generates a local point cloud map, removes ground points using methods such as RANSAC, and segments the point cloud data containing the target vehicle using point cloud clustering methods such as DBSCAN, defining it as a local map. Afterwards, structural registration can be performed. Specifically, structural descriptors and distribution descriptors are generated from the local map, and their similarity is calculated with predefined sub-topological regions. For example, by comparing the differences in descriptors (which can train a classifier), or by calculating the relative pose with the point cloud of the sub-topological region, several possible target regions are obtained (which are multi-peaked distributions, generally with ambiguity in left-right information due to symmetry), while simultaneously acquiring the model's global pose. Due to symmetry, a topological path is selected through the 3D connectivity graph. Based on the positional pattern, it moves to the next topological node. After acquiring the initial target point pose, it advances along the ordered topological sequence through the 3D connectivity graph, using the positional pattern, which only requires wheel odometer data. When the mobile charging device moves to the next topology node, the relative pose of the current observation distance model can be estimated through the structure registration algorithm and the ICP algorithm. At this time, the relative pose of the two topology nodes can be obtained by the wheel odometer, and the relative pose constraint is defined as an edge. Figure 18 A triangular region. Least squares optimization can be performed on multiple triangle sides to minimize the cumulative worst-case scenario. Since the model is a unified whole, each optimization adjusts the global pose of the model.

[0226] This application provides a charging port positioning method, the method comprising: acquiring multiple first point clouds, multiple point cloud sets, and the positional relationship between the point cloud sets, wherein the multiple first point clouds are point clouds collected by sensors on a mobile charging device from a first local area of ​​a target vehicle, each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle, the multiple point cloud sets include a first point cloud set, the first point cloud set being a point cloud set pre-constructed for a charging port area; based on the similarity condition between the multiple first point clouds and a target point cloud set in the multiple point clouds, obtaining a first pose of the mobile charging device by point cloud registration between the multiple first point clouds and the target point cloud set, wherein the target point cloud set is not a first point cloud set; determining target displacement information based on the first pose and the positional relationship between the first point cloud set and the target point cloud set, the target displacement information being used to guide the mobile charging device to move from the first pose to the charging port area. In the above manner, multiple point clouds provide prior information for the search direction of the mobile charging device. Point cloud registration based on multiple first point clouds and target point clouds that meet the similarity conditions in multiple point clouds can calculate the pose of the mobile charging device relative to the target vehicle. Even if the charging port is not visible at the location of the mobile charging device (or the charging port is blocked by other obstacles), since the positional relationship between multiple point clouds (including the first point cloud of the charging port) has been generated in advance, the mobile charging device can be guided to the charging port area based on the pose of the mobile charging device and the relative positional relationship between the target point cloud and the first point cloud.

[0227] Reference Figure 21 , Figure 21 A charging port positioning device provided in the embodiments of this application, such as Figure 21 As shown, the device 2100 may include:

[0228] The acquisition module 2101 is used to acquire multiple first point clouds, multiple point cloud sets, and the positional relationship between each point cloud set. The multiple first point clouds are point clouds collected by sensors on the mobile charging device from a first local area of ​​the target vehicle. Each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle. The multiple point cloud sets include the first point cloud set, which is a point cloud set pre-constructed for the charging port area.

[0229] For a detailed description of the acquisition module 2101, please refer to the description of step 701 in the above embodiment, which will not be repeated here.

[0230] The pose determination module 2102 is used to obtain the first pose of the mobile charging device by registering the multiple first point clouds and the target point cloud set in the multiple point cloud sets according to the similarity condition. The target point cloud set is not the first point cloud set.

[0231] For a detailed description of the pose determination module 2102, please refer to the description of step 702 in the above embodiment, which will not be repeated here.

[0232] The navigation module 2103 is used to determine target displacement information based on the first pose and the positional relationship between the first point set and the target point set. The target displacement information is used to guide the mobile charging device to move from the first pose to the charging port area.

[0233] For a detailed description of the navigation module 2103, please refer to the description of step 703 in the above embodiments, which will not be repeated here.

[0234] Among them, multiple point cloud sets provide prior information for the search direction of the mobile charging device. Point cloud registration based on multiple first point clouds and target point clouds that meet the similarity conditions in multiple point clouds can calculate the pose of the mobile charging device relative to the target vehicle. Even if the charging port is not visible at the location of the mobile charging device (or the charging port is blocked by other obstacles), since the positional relationship between multiple point cloud sets (including the first point cloud set of the charging port) is generated in advance, the mobile charging device can be guided to the charging port area based on the pose of the mobile charging device and the relative positional relationship between the target point cloud set and the first point cloud set.

[0235] In one possible implementation, the plurality of point clouds further includes a second point cloud, which is a point cloud pre-constructed for a second local region, the second local region being located between the first local region and the charging port region;

[0236] The navigation module is specifically used for:

[0237] Based on the first pose and the positional relationship between the second point set and the target point set, the first displacement information in the target displacement information is determined. The first displacement information is used to guide the mobile charging device to move from the first pose to the second local area.

[0238] In one possible implementation, the acquisition module is further configured to:

[0239] Multiple second point clouds and pose change information are acquired. The multiple second point clouds are point clouds collected by the sensor on the second local area of ​​the target vehicle after the mobile charging device moves to the second local area based on the first displacement information. The pose change information is the pose change of the mobile charging device from the first pose to the second local area.

[0240] The pose determination module is also used for:

[0241] The second pose of the sensor is obtained based on the point cloud registration between the second point cloud set and the plurality of second point clouds;

[0242] Using the first pose, the second pose, the pose change information, and the positional relationship between the second point set and the target point set as closed-loop constraints, the second pose is optimized to obtain the optimized second pose.

[0243] In this system, each time the mobile charging device moves, it can perform closed-loop optimization of its current pose based on the positional relationships between pre-built point clouds and the pose change information it has collected, thereby improving the accuracy of pose determination.

[0244] In one possible implementation, the pose determination module is further configured to: before the point cloud registration between the plurality of first point clouds and the target point cloud set, based on the fact that the similarity between the plurality of first point clouds and M point cloud sets in the plurality of point clouds is greater than a threshold, obtain a plurality of third point clouds, where M is an integer greater than 1, the plurality of third point clouds are point clouds collected by the sensor from a third local region of the target vehicle, the similarity between the plurality of third point clouds and the third point cloud sets in the plurality of point clouds is greater than a threshold, and the positional relationship between the third local region and the first local region is a first positional relationship;

[0245] Based on the fact that the positional relationship between the third point cloud set and the target point cloud set in the M point cloud sets is the same as the first positional relationship, it is determined that the plurality of first point clouds and the target point cloud sets in the plurality of point cloud sets satisfy the similarity condition.

[0246] When a unique point cloud set satisfying the similarity condition cannot be matched, the mobile charging device can be guided from the first position to the second position. Although there are multiple point cloud sets with high similarity among the multiple first point clouds, the scope can be narrowed down to the target point cloud set among the multiple point cloud sets based on the relative positional relationship between the local regions corresponding to these multiple point cloud sets and the surrounding local regions. For example, when the first local region is the corner area of ​​the right front of the vehicle, when performing similarity matching on multiple first point clouds, the point cloud sets of the multiple point clouds and the four corner areas on the target vehicle all have a similarity greater than the threshold with the multiple first point clouds. However, the point cloud set of the adjacent area of ​​the first local region in the clockwise direction towards the vehicle body is the point cloud set of the right headlight, and the adjacent area of ​​other corner areas in the clockwise direction towards the vehicle body is not the right headlight (the adjacent area of ​​the left rear corner in the clockwise direction towards the vehicle body is the left door, the adjacent area of ​​the left front corner in the clockwise direction towards the vehicle body is the side surface of the front of the vehicle, or the license plate area of ​​the front of the vehicle, and the adjacent area of ​​the right rear corner towards the vehicle body is the right headlight). The adjacent area in the clockwise direction is the side surface of the rear of the vehicle (or the license plate area of ​​the rear of the vehicle). Therefore, the mobile charging device can be guided to move a certain distance in a certain direction to the second position and collect multiple third point clouds in the third local area. Based on the similarity matching between the multiple third point clouds and the multiple point cloud sets, it is possible to determine which point cloud set is accurate from the multiple point cloud sets. Furthermore, based on the fact that the positional relationship between the third point cloud set and the target point cloud set in the multiple point cloud sets is the same as the first positional relationship, it can be determined that the multiple first point clouds and the target point cloud set in the multiple point cloud sets meet the similarity condition.

[0247] It should be understood that the selection range of a portion of the point cloud set can be narrowed down based on the height information of multiple first point clouds and the ground (generally, the point cloud located at the front of the vehicle will be at a lower height relative to the ground than the point cloud located at the rear of the vehicle). In other words, based on the point cloud distribution, the height above the ground, and the relative relationship between the surrounding area (e.g., the third local area) and the first local area, the point cloud set (target point cloud set) that matches the similarity of multiple first point clouds can be uniquely determined.

[0248] In one possible implementation, the third local area is a door area, a headlight area, a wheel area, or a license plate area.

[0249] In one possible implementation, the first local region is the corner region of the vehicle body, which is formed by two surfaces on the target vehicle that are perpendicular to the ground and mutually perpendicular, and the plurality of first point clouds are the point clouds of the corner region of the vehicle body on a preset height plane.

[0250] In one possible implementation, the first local region is the side surface region of the target vehicle facing forward or backward, and the plurality of first point clouds are point clouds of the side surface region on a preset height plane.

[0251] In one possible implementation, the point cloud range of some areas in the aforementioned regions with significant features is very small (due to the small area of ​​the region itself), and the semantics of the point cloud of the local region (specifically, which part of the target vehicle it is) cannot be directly determined. However, the corner area of ​​the front of the vehicle or the side surface area facing the front or rear of the vehicle, when having significant features, usually appears in the field of view of the sensor due to the large area of ​​the region. Using it as the initial point cloud registration object can improve the success rate of matching.

[0252] In one possible implementation, the plurality of first point clouds are point clouds collected by the sensor of a first local area on the target vehicle when the mobile charging device is in a first position, and the plurality of third point clouds are point clouds collected by the sensor of a third local area on the target vehicle when the mobile charging device is in a second position, wherein the second position is different from the first position.

[0253] The navigation module is also used for:

[0254] Guide the mobile charging device from the first position to the second position.

[0255] The first position and the second position are the positions around the target vehicle, and the second position can be close to the first position (e.g., 20cm, 30cm). This distance can be preset, as long as the sensor of the mobile charging device can collect other local areas (third local areas) besides the first local area at the second position, and is not limited to the area where the second position is located.

[0256] In one possible implementation, the similarity includes: the similarity of the point cloud distribution and / or the similarity of the ground elevation of the point cloud.

[0257] In one possible implementation, the first pose is the relative pose between the sensor and the target vehicle.

[0258] The following describes a charging port positioning device provided in an embodiment of this application. Please refer to [link to relevant documentation]. Figure 22 , Figure 22 This is a schematic diagram of a charging port positioning device provided in an embodiment of this application. Specifically, the charging port positioning device 2200 includes: a receiver 2201, a transmitter 2202, a processor 2203, and a memory 2204 (wherein the number of processors 2203 in the charging port positioning device 2200 can be one or more). Figure 22(Taking a processor as an example), processor 2203 may include application processor 22031 and communication processor 22032. In some embodiments of this application, receiver 2201, transmitter 2202, processor 2203 and memory 2204 may be connected via bus or other means.

[0259] Memory 2204 may include read-only memory and random access memory, and provides instructions and data to processor 2203. A portion of memory 2204 may also include non-volatile random access memory (NVRAM). Memory 2204 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0260] Processor 2203 controls the operation of the radar system (including antenna, receiver 2201, and transmitter 2202). In specific applications, the various components of the radar system are coupled together through a bus system, which may include a data bus, as well as a power bus, control bus, and status signal bus. However, for clarity, all buses are referred to as the bus system in the diagram.

[0261] The charging port positioning method disclosed in the above embodiments of this application ( Figure 7The processor 2203 (as shown) can be applied to or implemented by the processor 2203. The processor 2203 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 2203 or by instructions in software form. The processor 2203 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 2203 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 2204. Processor 2203 reads the information in memory 2204 and, in conjunction with its hardware, completes the steps of the charging port positioning method provided in the above embodiments.

[0262] Receiver 2201 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the radar system. Transmitter 2202 can be used to output digital or character information through the first interface; transmitter 2202 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group.

[0263] This application also provides a computer program product that, when run on a computer, causes the computer to execute the charging port positioning method described in the above embodiments.

[0264] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to execute the charging port positioning method described in the above embodiments.

[0265] The charging port positioning device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the execution device to execute the image enhancement method described in the above embodiment, or to cause the chip in the training device to execute the image enhancement method described in the above embodiment. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, such as random access memory (RAM).

[0266] For details, please refer to Figure 23 , Figure 23 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 230. The NPU 230 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 2303, which is controlled by the controller 2304 to extract matrix data from the memory and perform multiplication operations.

[0267] In some implementations, the arithmetic circuit 2303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 2303 is a two-dimensional pulsating array. The arithmetic circuit 2303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2303 is a general-purpose matrix processor.

[0268] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2308.

[0269] Unified memory 2306 is used to store input and output data. Weight data is directly transferred to weight memory 2302 via direct memory access controller (DMAC) 2305. Input data is also transferred to unified memory 2306 via DMAC.

[0270] BIU stands for Bus Interface Unit 2310, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 2309.

[0271] The Bus Interface Unit (BIU) 2310 is used by the instruction fetch memory 2309 to fetch instructions from external memory, and also by the memory access controller 2305 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0272] The DMAC is mainly used to move input data from external memory DDR to unified memory 2306, or to weight data to weight memory 2302, or to input data to input memory 2301.

[0273] The vector computation unit 2307 includes multiple arithmetic processing units that, when necessary, further process the output of the computation circuit, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0274] In some implementations, the vector computation unit 2307 can store the processed output vector in the unified memory 2306. For example, the vector computation unit 2307 can apply linear and / or nonlinear functions to the output of the computation circuit 2303, such as performing linear interpolation on feature planes extracted by convolutional layers, or accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 2307 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 2303, for example, for use in subsequent layers of the neural network.

[0275] The instruction fetch buffer 2309 connected to the controller 2304 is used to store the instructions used by the controller 2304;

[0276] The unified memory 2306, input memory 2301, weighted memory 2302, and instruction fetch memory 2309 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0277] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of programs related to the charging port positioning method described in the above embodiments.

[0278] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0279] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0280] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0281] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A method for locating a charging port, characterized in that, The method includes: Multiple first point clouds, multiple point cloud sets, and the positional relationships between the point cloud sets are obtained. The multiple first point clouds are point clouds collected by sensors on the mobile charging device from a first local area of ​​the target vehicle. Each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle. The multiple point cloud sets include the first point cloud set, which is a point cloud set pre-constructed for the charging port area. The multiple point cloud sets have a pre-constructed fixed positional relationship, which is the spatial relative positional relationship of different local areas on the target vehicle. Based on the fact that the plurality of first point clouds and the target point cloud set in the plurality of point clouds meet the similarity condition, the first pose of the mobile charging device is obtained by point cloud registration between the plurality of first point clouds and the target point cloud set. The first pose is the relative pose between the sensor and the target vehicle, and the target point cloud set is not the first point cloud set. Based on the first pose and the fixed positional relationship between the first point set and the target point set, target displacement information is determined. The target displacement information is used to guide the mobile charging device to move from the first pose to the charging port area.

2. The method according to claim 1, characterized in that, The plurality of point clouds also includes a second point cloud, which is a point cloud pre-constructed for a second local region, the second local region being located between the first local region and the charging port region; The step of determining the target displacement information based on the first pose and the positional relationship between the first point set and the target point set includes: Based on the first pose and the positional relationship between the second point set and the target point set, the first displacement information in the target displacement information is determined. The first displacement information is used to guide the mobile charging device to move from the first pose to the second local area.

3. The method according to claim 2, characterized in that, After determining the first displacement information in the target displacement information, the method further includes: Multiple second point clouds and pose change information are acquired. The multiple second point clouds are point clouds collected by the sensor on the second local area of ​​the target vehicle after the mobile charging device moves to the second local area based on the first displacement information. The pose change information is the pose change of the mobile charging device from the first pose to the second local area. The second pose of the sensor is obtained based on the point cloud registration between the second point cloud set and the plurality of second point clouds; Using the first pose, the second pose, the pose change information, and the positional relationship between the second point set and the target point set as closed-loop constraints, the second pose is optimized to obtain the optimized second pose.

4. The method according to claim 1, characterized in that, Before performing point cloud registration between the plurality of first point clouds and the target point cloud set, the method further includes: Based on the fact that the similarity between the plurality of first point clouds and M point cloud sets in the plurality of point clouds is greater than a threshold, a plurality of third point clouds are obtained, where M is an integer greater than 1. The plurality of third point clouds are point clouds collected by the sensor from the third local region of the target vehicle. The similarity between the plurality of third point clouds and the third point cloud sets in the plurality of point clouds is greater than a threshold. The positional relationship between the third local region and the first local region is the first positional relationship. Based on the fact that the positional relationship between the third point cloud set and the target point cloud set in the M point cloud sets is the same as the first positional relationship, it is determined that the plurality of first point clouds and the target point cloud sets in the plurality of point cloud sets satisfy the similarity condition.

5. The method according to claim 4, characterized in that, The first local area is the corner area of ​​the vehicle body, which is formed by two surfaces on the target vehicle that are perpendicular to the ground and mutually perpendicular. The plurality of first point clouds are the point clouds of the corner area of ​​the vehicle body on a preset height plane.

6. The method according to claim 4, characterized in that, The first local area is the side surface area of ​​the target vehicle facing the front or rear of the vehicle, and the plurality of first point clouds are point clouds of the side surface area on a preset height plane.

7. The method according to any one of claims 4 to 6, characterized in that, The third local area is the door area, headlight area, wheel area, or license plate area.

8. The method according to claim 4, characterized in that, The plurality of first point clouds are point clouds collected by the sensor on a first local area of ​​the target vehicle when the mobile charging device is in a first position, and the plurality of third point clouds are point clouds collected by the sensor on a third local area of ​​the target vehicle when the mobile charging device is in a second position, wherein the second position is different from the first position. Before acquiring multiple third point clouds, the method further includes: Guide the mobile charging device from the first position to the second position.

9. The method according to claim 4, characterized in that, The similarity includes: Similarity of point cloud distribution and / or similarity of the ground elevation of the point clouds.

10. A charging port positioning device, characterized in that, The device includes: The acquisition module is used to acquire multiple first point clouds, multiple point cloud sets, and the positional relationship between each point cloud set. The multiple first point clouds are point clouds collected by sensors on the mobile charging device from a first local area of ​​the target vehicle. Each point cloud set is a point cloud set pre-constructed for a local area of ​​the target vehicle. The multiple point cloud sets include the first point cloud set, which is a point cloud set pre-constructed for the charging port area. The multiple point cloud sets have a pre-constructed fixed positional relationship, which is the spatial relative positional relationship of different local areas on the target vehicle. The pose determination module is used to obtain the first pose of the mobile charging device by registering the multiple first point clouds and the target point cloud set in the multiple point clouds with a similarity condition. The first pose is the relative pose between the sensor and the target vehicle, and the target point cloud set is not the first point cloud set. The navigation module is used to determine target displacement information based on the first pose and the fixed positional relationship between the first point set and the target point set. The target displacement information is used to guide the mobile charging device to move from the first pose to the charging port area.

11. The apparatus according to claim 10, characterized in that, The plurality of point clouds also includes a second point cloud, which is a point cloud pre-constructed for a second local region, the second local region being located between the first local region and the charging port region; The navigation module is specifically used for: Based on the first pose and the positional relationship between the second point set and the target point set, the first displacement information in the target displacement information is determined. The first displacement information is used to guide the mobile charging device to move from the first pose to the second local area.

12. The apparatus according to claim 11, characterized in that, The acquisition module is also used for: Multiple second point clouds and pose change information are acquired. The multiple second point clouds are point clouds collected by the sensor on the second local area of ​​the target vehicle after the mobile charging device moves to the second local area based on the first displacement information. The pose change information is the pose change of the mobile charging device from the first pose to the second local area. The pose determination module is also used for: The second pose of the sensor is obtained based on the point cloud registration between the second point cloud set and the plurality of second point clouds; Using the first pose, the second pose, the pose change information, and the positional relationship between the second point set and the target point set as closed-loop constraints, the second pose is optimized to obtain the optimized second pose.

13. The apparatus according to claim 10, characterized in that, The pose determination module is further configured to: before performing point cloud registration between the plurality of first point clouds and the target point cloud set, obtain a plurality of third point clouds based on the fact that the similarity between the plurality of first point clouds and M point cloud sets in the plurality of point clouds is greater than a threshold, wherein M is an integer greater than 1, the plurality of third point clouds are point clouds collected by the sensor from the third local region of the target vehicle, the similarity between the plurality of third point clouds and the third point cloud sets in the plurality of point clouds is greater than a threshold, and the positional relationship between the third local region and the first local region is a first positional relationship; Based on the fact that the positional relationship between the third point cloud set and the target point cloud set in the M point cloud sets is the same as the first positional relationship, it is determined that the plurality of first point clouds and the target point cloud sets in the plurality of point cloud sets satisfy the similarity condition.

14. The apparatus according to claim 13, characterized in that, The first local area is the corner area of ​​the vehicle body, which is formed by two surfaces on the target vehicle that are perpendicular to the ground and mutually perpendicular. The plurality of first point clouds are the point clouds of the corner area of ​​the vehicle body on a preset height plane.

15. The apparatus according to claim 13, characterized in that, The first local area is the side surface area of ​​the target vehicle facing the front or rear of the vehicle, and the plurality of first point clouds are point clouds of the side surface area on a preset height plane.

16. The apparatus according to any one of claims 13 to 15, characterized in that, The third local area is the door area, headlight area, wheel area, or license plate area.

17. The apparatus according to claim 13, characterized in that, The plurality of first point clouds are point clouds collected by the sensor on a first local area of ​​the target vehicle when the mobile charging device is in a first position, and the plurality of third point clouds are point clouds collected by the sensor on a third local area of ​​the target vehicle when the mobile charging device is in a second position, wherein the second position is different from the first position. The navigation module is also used for: Guide the mobile charging device from the first position to the second position.

18. The apparatus according to claim 13, characterized in that, The similarity includes: Similarity of point cloud distribution and / or similarity of the ground elevation of the point clouds.

19. A computer-readable storage medium, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method according to any one of claims 1 to 9.

20. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 9.

21. A mobile charging device, characterized in that, The mobile charging device includes a sensor, a driving device, one or more processors, and a memory; wherein the memory stores computer-readable instructions. The one or more processors read the computer-readable instructions to control the sensor and the drive device, and perform the method as described in any one of claims 1 to 9.

Images