Docking control method and device of mobile equipment, equipment and storage medium

By using image acquisition equipment in the automatic guide car to obtain point cloud images of the pallets and match reference images in the image library, the problem of high computing resources consumption during docking in the prior art is solved, and accurate pallets and carts are realized Docking.

CN118981208APending Publication Date: 2024-11-19GUANGDONG LEAD INTELLIGENT LOGISTICS TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411045880.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

During the docking process of existing automatic guided trolleys and cargo pallets, more computing resources are needed to determine the relative position of the pallet and the trolley.

Method used

The image acquisition device of the mobile device acquires the sampled image of the tray, uses color information and depth information to generate a point cloud image, and determines a reference image similar to the point cloud image in the preset image library, thereby controlling the position of the car and reducing the position of the car. The consumption of computing resources.

Benefits of technology

It realizes accurate positioning of the relative position of the pallet and the mobile device, effectively reducing the consumption of computing resources, and improving docking accuracy and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981208A_ABST
    Figure CN118981208A_ABST
Patent Text Reader

Abstract

The invention relates to a docking control method, device and equipment of mobile equipment and a storage medium, and belongs to the technical field of automatic guide trolleys, and the method comprises the steps: obtaining a sampling image related to a tray through image collection equipment of the mobile equipment; obtaining a point cloud image about the tray according to the color information and the depth information of the sampling image; determining a first reference image in reference images in a preset image library; wherein the feature point cloud of the point cloud image is aligned with the feature point cloud of the first reference image after set transformation processing; and controlling the pose of the mobile device according to the first pose information reflected by the first reference image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of automatic guided vehicles, and more specifically, to a docking control method, apparatus, device and storage medium for a mobile device. Background Art

[0002] With the rapid development of the logistics industry, Automated Guided Vehicles (AGVs) can be used as the main logistics transportation vehicles. At present, the existing AGVs can be docked with pallets of goods by identifying the two-dimensional image of the pallet to determine the relative position of the pallet and the AGV, which consumes a lot of computing resources. Summary of the invention

[0003] One purpose of the embodiments of the present disclosure is to provide a new technical solution for docking control of a mobile device.

[0004] According to a first aspect of the present disclosure, a docking control method for a mobile device is provided, the method comprising:

[0005] Acquire a sample image of the tray through an image acquisition device of the mobile device;

[0006] Obtaining a point cloud image of the pallet according to the color information and depth information of the sampled image;

[0007] Determine a first reference image from each reference image in a preset image library; wherein the feature point cloud of the point cloud image is aligned with the feature point cloud of the first reference image after a set transformation process;

[0008] The posture of the mobile device is controlled according to the first posture information reflected by the first reference image.

[0009] Optionally, obtaining a point cloud image of the pallet according to the color information and the depth information of the sampled image includes:

[0010] Inputting the sampled image into a preset segmentation model to extract a target area including the tray;

[0011] A point cloud image of the pallet is obtained according to the target area and the set internal parameter calibration.

[0012] Optionally, after obtaining a point cloud image of the pallet according to the color information and depth information of the sampled image, the method further includes:

[0013] The point cloud image is subjected to data processing to obtain a processed point cloud image; wherein the data processing includes outlier denoising, downsampling, and removal of point cloud planes whose heights are lower than a set height threshold.

[0014] Optionally, before determining the first reference image from among the reference images in the preset image library, the method further includes:

[0015] Determining an offset angle of the pallet relative to the mobile device according to the outline of the pallet reflected by the point cloud image;

[0016] The step of determining a first reference image from among the reference images in the preset image library comprises:

[0017] Selecting, from each reference image in a preset image library, a second reference image whose offset angle is within a set angle threshold;

[0018] Registering the second reference image and the point cloud image to obtain a registration result;

[0019] The registration result is determined to be a second reference image that meets a set convergence condition and is used as the first reference image.

[0020] Optionally, registering the second reference image and the point cloud image to obtain a registration result includes:

[0021] According to the set rigid body transformation parameters, the point cloud image is subjected to rigid body transformation processing;

[0022] The minimized residual square sum between the second reference image after the rigid body transformation processing and the point cloud image is determined and used as the registration result.

[0023] Optionally, before controlling the posture of the mobile device according to the first posture information reflected by the first reference image, the method further includes:

[0024] Determining rigid body transformation parameters corresponding to the first reference image;

[0025] The controlling the posture of the mobile device according to the first posture information reflected by the first reference image includes:

[0026] Obtaining second posture information of the mobile device relative to the tray in a camera coordinate system according to the first posture information reflected by the first reference image and the rigid body transformation parameters corresponding to the first reference image;

[0027] According to the set conversion relationship and the second posture information, third posture information of the mobile device relative to the tray in the vehicle coordinate system is obtained; wherein the conversion relationship reflects that different points in the camera coordinate system correspond to different points in the vehicle coordinate system;

[0028] The posture of the mobile device is controlled according to the third posture information.

[0029] According to a second aspect of the present disclosure, there is also provided a docking control device for a mobile device, comprising:

[0030] An acquisition module, used for acquiring a sample image of the tray through an image acquisition device of a mobile device;

[0031] An obtaining module, used for obtaining a point cloud image of the pallet according to the color information and depth information of the sampled image;

[0032] A determination module, configured to determine a first reference image similar to the point cloud image from among the reference images in a preset image library;

[0033] A control module is used to control the posture of the mobile device according to the first posture information reflected by the first reference image.

[0034] According to a third aspect of the present disclosure, an electronic device is also provided, including a processor, wherein the processor is connected to a memory, and the memory is used to store a computer program; the processor is used to execute the computer program to implement the method described in the first aspect of the present disclosure.

[0035] According to a fourth aspect of the present disclosure, a mobile device is further provided, comprising the mobile device as described in the second aspect or the electronic device as described in the third aspect.

[0036] According to a fifth aspect of the present disclosure, a computer storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0037] A beneficial effect of the embodiments of the present disclosure is that the docking control method of the mobile device provided by the present invention can obtain a sampled image about the pallet through an image acquisition device, use the color information and depth information of the sampled image to obtain a point cloud image of the pallet, and determine a reference image similar to the point cloud image from the image library, and use the first pose information of the reference image to control the mobile device to move or turn. Through the color information and depth information, a more accurate point cloud image can be obtained, and the first reference image similar to the point cloud image can be screened to control the mobile device, effectively reducing the consumption of computing resources required to calculate the relative position of the pallet and the mobile device.

[0038] Other features and advantages of the embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0040] Figure 1 is a schematic diagram of the composition structure of a docking system of a mobile device to which a docking control method of a mobile device according to an embodiment can be applied;

[0041] Figure 2 is a schematic structural diagram of a tray according to an embodiment;

[0042] Figure 3 is a flow chart of a method for controlling docking of a mobile device according to an embodiment;

[0043] Figure 4 is a block diagram of a docking control device for a mobile device according to an embodiment;

[0044] Figure 5 is a schematic diagram of the hardware structure of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0045] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0046] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0047] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0048] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0049] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] <System Embodiment>

[0051] Figure 1 FIG. 1 is a schematic diagram of the structure of a docking system of a mobile device to which a docking control method of a mobile device according to an embodiment can be applied. Figure 1 As shown, the docking control system of the mobile device includes a mobile device 10 and a tray 20 .

[0052] The mobile device 10 may be an automated guided vehicle (AGV), and the mobile device 10 is equipped with a corresponding image acquisition device 11, such as an RGB-D camera.

[0053] The above pallet 20 can be used to store goods, such as Figure 2 As shown, the tray 20 may include a storage platform 21 and legs 22, wherein the legs 22 are disposed at the bottom of the storage platform 21, and the top of the storage platform 21 is used to store goods.

[0054] As used in the embodiments of the present disclosure, the memory of the mobile device 10 is used to store a computer program, which is used to control the processor of the mobile device 10 to operate according to the docking control method of the mobile device of any embodiment. A technician can design a computer program according to the scheme of the embodiments of the present disclosure. How the computer program controls the processor to operate is well known in the art, so it will not be described in detail here.

[0055] <Method Example>

[0056] Figure 2 is a flow chart of a method for controlling docking of a mobile device according to an embodiment. Figure 1 The mobile device 10.

[0057] like Figure 2 As shown, the docking control method of the mobile device of this embodiment may include the following steps S310 to S340:

[0058] Step S310: acquiring a sample image of the tray through an image acquisition device of a mobile device.

[0059] In the present application, the image acquisition device may be an RGB-D camera, which may output a color map and a depth map as a sampled image, or may output a fused image of the color map and the depth map as a sampled image.

[0060] In the present application, the mobile device is, for example, the above-mentioned automatic guided vehicle. The automatic guided vehicle can be equipped with a navigation system, and the navigation system can provide feedback to the automatic guided vehicle on surrounding pallets, production lines, equipment, etc. With the cooperation of the navigation system and the image acquisition device, the mobile device can obtain sample images of the pallet.

[0061] Step S320: obtaining a point cloud image of the pallet according to the color information and depth information of the sampled image.

[0062] By sampling the color information and depth information of the image, a point cloud that can represent the length, width, height and depth of the pallet is determined to form a point cloud image of the pallet.

[0063] In some embodiments, step S320 may include the following steps S3201 and S3202:

[0064] Step S3201: input the sampled image into a preset segmentation model to extract the target area including the tray.

[0065] In this application, the segmentation model uses the YOLOv8-seg model to segment the pallet. In addition, the YOLOv8-seg instance segmentation model is pre-trained using a data sample set, and the pallet instances are labeled and hyperparameters are adjusted. The sampled image is input into the YOLOv8-seg model, and a plurality of bounding boxes of image instances containing pallets can be obtained. According to different pallet selection strategies, such as the largest, the middle, the highest confidence, etc., the ROI area of ​​the pallet, i.e., the target area, is cropped using the bounding box. In other words, by setting the segmentation model, the high speed of object detection and the high precision of segmentation are combined, which significantly improves the recognition accuracy of the pallet boundary, and can distinguish the pallet from the surrounding environment in a complex background.

[0066] Step S3202, calibrating according to the target area and the set internal parameters to obtain a point cloud image of the pallet.

[0067] In the present application, the set internal parameter calibration is used to estimate the mapping of the three-dimensional sampling image output by the image acquisition device into a two-dimensional point cloud image.

[0068] In this application, the target area is mapped into the depth map, and then the intrinsic parameters of the IR lens of the RGB-D camera (f x ,f y ,c x ,c y ), the point cloud image Z of the ROI area can be calculated and constructed:

[0069]

[0070] In other words, the target area is extracted through the color map, and then the depth map is used to determine the point cloud in the target area that can represent the pallet, that is, the point cloud image, which effectively reduces the computational burden and improves the operating efficiency of the mobile device.

[0071] In some embodiments, after step S320, the following step S3203 is also included:

[0072] Step S3203, performing data processing on the point cloud image to obtain a processed point cloud image; wherein the data processing includes outlier denoising, downsampling, and removing point cloud planes whose heights are lower than a set height threshold.

[0073] In this application, existing outlier denoising algorithms, such as statistical filtering algorithms, radius filtering algorithms, etc., are used to reduce outliers or noise points in point cloud images that significantly deviate from the normal range, so as to improve the efficiency of data processing and storage of point cloud images.

[0074] In the present application, the existing downsampling algorithms, such as voxel grid downsampling or farthest point sampling, are used to reduce the data volume of the point cloud image and improve the efficiency of data processing and storage of the point cloud image.

[0075] In this application, in order to remove the ground or pallet loading plane corresponding to the point cloud image, the normal vector of each contour line in the point cloud image is determined, and the contour line whose normal vector component on the Y axis is greater than the set value is determined, that is, it is considered to be a point cloud plane close to the horizontal plane, and the point cloud plane in the point cloud image is removed to further improve the efficiency of data processing and storage of the point cloud image. The set value is, for example, 0.7, 0.8 or 0.9, etc., which is not limited here.

[0076] Step S330, determining a first reference image from each reference image in a preset image library; wherein the feature point cloud of the point cloud image is aligned with the feature point cloud of the first reference image after a set transformation process.

[0077] In the present application, the image library pre-stores a plurality of point cloud images of the pallet at different angles and serves as reference images.

[0078] In some embodiments, before step S330, the method further includes the following step S329:

[0079] Step S329: Determine the offset angle of the pallet relative to the mobile device according to the outline of the pallet reflected by the point cloud image.

[0080] In the present application, the offset angle α of the tray relative to the mobile device can be determined by determining the normal vector of the outline of the tray.

[0081] In some examples, such as Figure 2 As shown, the mobile device can use the middle point P of the middle rectangular block on the front face of the pallet as the representative point of the pallet. Figure 1In the two-dimensional map navigation shown, the pallet can be displayed as a rectangle in the two-dimensional map, and in the two-dimensional map, the position of the pallet relative to the mobile device can be determined by the X-axis information, the Z-axis information and the offset angle θ, without the Y-axis information, for example, Figure 1 As shown, the position of tray 20a is (0, 0, 20), the position of tray 20b is (0, 0, 0), and the position of tray 20c is (0, 0, -20).

[0082] Based on this, step S330 may include the following steps S3301 to S3303:

[0083] Step S3301 : selecting, from among the reference images in the preset image library, a second reference image whose offset angle is within a set angle threshold.

[0084] In the present application, the angle threshold is set to, for example, 5°, 10° or 20°, etc., which is not limited here.

[0085] Taking setting the angle threshold to 10° as an example, the second reference image in the interval of [α-10°, α+10°] is selected from the image library.

[0086] Step S3302: align the second reference image and the point cloud image to obtain a alignment result.

[0087] In this application, the feature points and normal vectors of the pallet's contour in the point cloud image are extracted to form an initial closest point pair and associate with the points in the second reference image. In addition, the registration error between the second reference image and the point cloud image is minimized through iterative optimization, and the point cloud image and the second reference image are aligned to obtain multiple registration errors, i.e., registration results.

[0088] Step S3303: Determine that the registration result is a second reference image that meets the set convergence condition and use it as the first reference image.

[0089] When the values ​​of multiple registration errors are less than the set value, the registration result is determined to meet the set convergence condition, and the second reference image that meets the set convergence condition is used as the first reference image. In other words, the registration result can be a transformation method that reduces the distance between the feature point cloud of the second reference image and the feature point cloud of the point cloud image by translating or rotating the point cloud image, and determines the reference image corresponding to the minimum distance among these second reference images and uses it as the first reference image. At the same time, after determining the first reference image, the transformation parameters of the point cloud image to the first reference image can be obtained.

[0090] In other words, by extracting the feature points of the point cloud image, forming the closest point pairs with the points in the second reference image, and determining the registration error between the second reference image and the point cloud image, the first reference image whose registration error meets the convergence condition is determined, which can effectively improve the accuracy of the determined first reference image.

[0091] In some embodiments, step S3302 may include the following steps S410 and S420:

[0092] Step S410, performing rigid body transformation processing on the point cloud image according to the set rigid body transformation parameters.

[0093] In the present application, the rigid body transformation parameter is the parameter T corresponding to the translation or rotation, so as to translate or rotate the point cloud image. In other words, the transformation processing set for the feature point cloud of the point cloud image is to translate or rotate the point cloud image by the parameter T1 corresponding to the first reference image, so that the feature point cloud of the point cloud image is aligned with the feature point cloud of the first reference image.

[0094] Step S420, determining the minimized residual sum of squares between the second reference image after rigid body transformation processing and the point cloud image, and taking it as the registration result.

[0095] In this application, the expression for minimizing the sum of squares of the distance metric between the second reference image and the point cloud image, that is, minimizing the sum of squares of the residual error, is as follows:

[0096] min T ∑ i ‖p i -Tq i ‖ 2

[0097] In the formula, p i is the point of the second reference image, q i is the corresponding point of the point cloud image, T is the rigid body transformation parameter, so that the transformed Tq i , as close to p as possible i Furthermore, when the minimization of the residual sum of squares does not meet the convergence condition, one iteration will be continued, and in each iteration, the updated transformation parameter T will be calculated so that the above-mentioned minimization of the residual sum of squares is gradually reduced until the convergence condition is met.

[0098] In other words, the initial closest point pair is constructed through the three-dimensional point cloud registration algorithm (Generalized Iterative Closest Point, GICP), and the registration error is determined. When the registration error meets the set convergence condition, the first reference image is transformed to the best alignment state with the point cloud image through a rigid body transformation parameter to achieve accurate positioning of the pallet.

[0099] Step S340: Control the posture of the mobile device according to the first posture information reflected by the first reference image.

[0100] In the present application, the first pose information may be a relative angle and a relative distance of the mobile device relative to a reference object in the first reference image, wherein the reference object may be a tray.

[0101] In the present application, the posture of the mobile device may be the actual relative angle and actual relative distance of the mobile device relative to the tray. The mobile device may regulate the movement and / or steering of a driving component of the mobile device according to the relative angle and relative distance, so that the mobile device is facing the tray, and the docking end face of the mobile device can be brought close to the docking face of the tray for docking.

[0102] In some embodiments, before step S340, the method further includes the following step S339:

[0103] Step S339: determine the rigid body transformation parameters corresponding to the first reference image.

[0104] Based on this, step S340 may include the following steps S3401 to S3403:

[0105] Step S3401, obtaining second posture information of the mobile device relative to the tray in the camera coordinate system according to the first posture information reflected by the first reference image and the rigid body transformation parameters corresponding to the first reference image.

[0106] In this application, the rigid body transformation parameter corresponding to the first reference image is T b , the first pose information reflected by the first reference image is P, then the second pose information of the mobile device relative to the tray in the camera coordinate system is P c =T b *P.

[0107] Step S3402, according to the set conversion relationship and the second posture information, obtain the third posture information of the mobile device relative to the tray in the vehicle coordinate system; wherein the conversion relationship reflects that different points in the camera coordinate system correspond to different points in the vehicle coordinate system.

[0108] In this application, the conversion relationship set is the transformation relationship between the camera coordinate system and the vehicle body coordinate system T vc , the third pose information is P v =T vc *P c .

[0109] Step S3403: Control the posture of the mobile device according to the third posture information.

[0110] In this application, the third posture information P v Provide real-time feedback to mobile devices to guide them into storage and complete the task of picking up pallets.

[0111] In other words, through the set conversion relationship, the first posture information can be converted into the third posture information that conforms to the movement of the corresponding mobile device body, which can effectively improve the docking accuracy of the mobile device and the pallet.

[0112] <Equipment Example 1>

[0113] Figure 4 FIG. 1 is a block diagram of a docking control device for a mobile device according to an embodiment. Figure 4 As shown, the docking control device 400 of the mobile device may include:

[0114] An acquisition module 410 is used to acquire a sample image of the tray through an image acquisition device of a mobile device;

[0115] An obtaining module 420 is used to obtain a point cloud image of the pallet according to the color information and depth information of the sampled image;

[0116] The determination module 430 is used to determine a first reference image from each reference image in a preset image library; wherein the feature point cloud of the point cloud image is aligned with the feature point cloud of the first reference image after a set transformation process;

[0117] The control module 440 is used to control the posture of the mobile device according to the first posture information reflected by the first reference image.

[0118] Optionally, the obtaining module 420 is further used to input the sampled image into a preset segmentation model to extract a target area including the pallet; and obtain a point cloud image of the pallet according to the target area and the set internal parameter calibration.

[0119] Optionally, the docking control device 400 of the mobile device also includes an image acquisition module, which is used to perform data processing on the point cloud image to obtain a processed point cloud image; wherein the data processing includes outlier denoising, downsampling and removal of point cloud planes whose heights are lower than a set height threshold.

[0120] Optionally, the docking control device 400 for the mobile device further includes an angle determination module, which is used to determine the offset angle of the pallet relative to the mobile device according to the outline of the pallet reflected by the point cloud image;

[0121] The determination module 430 is also used to screen out a second reference image whose offset angle is within a set angle threshold from each reference image in a preset image library; align the second reference image and the point cloud image to obtain an alignment result; determine that the alignment result is a second reference image that meets the set convergence conditions and use it as the first reference image.

[0122] Optionally, the determination module 430 is further used to perform rigid body transformation processing on the point cloud image according to set rigid body transformation parameters; determine the minimized residual sum of squares between the second reference image after rigid body transformation processing and the point cloud image, and use it as the registration result.

[0123] Optionally, the docking control device 400 of the mobile device further includes a parameter determination module, which is used to determine the rigid body transformation parameters corresponding to the first reference image;

[0124] The control module 440 is also used to obtain second posture information of the mobile device relative to the tray in the camera coordinate system based on the first posture information reflected by the first reference image and the rigid body transformation parameters corresponding to the first reference image; obtain third posture information of the mobile device relative to the tray in the vehicle coordinate system based on the set conversion relationship and the second posture information; wherein the conversion relationship reflects that different points in the camera coordinate system correspond to different points in the vehicle coordinate system; and control the posture of the mobile device based on the third posture information.

[0125] The docking control device 400 of the mobile device may be Figure 1 A mobile device 10 is provided.

[0126] <Equipment Example 2>

[0127] Figure 5 is a schematic diagram of the hardware structure of an electronic device according to an embodiment.

[0128] like Figure 5 As shown, the electronic device 500 includes a processor 510 and a memory 520, wherein the memory 520 is used to store an executable computer program, and the processor 510 is used to execute a method such as any of the above method embodiments under the control of the computer program.

[0129] The electronic device 500 may be Figure 1 A mobile device 10 is provided.

[0130] Each module of the above-mentioned docking control device 400 for a mobile device may be implemented by the processor 510 in this embodiment executing a computer program stored in the memory 520, or may be implemented by other structures, which is not limited here.

[0131] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0132] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0133] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0134] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0135] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0137] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0138] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0139] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for controlling docking of a mobile device, characterized in that: The method comprises: Acquire a sample image of the tray through an image acquisition device of the mobile device; Obtaining a point cloud image of the pallet according to the color information and depth information of the sampled image; Determine a first reference image from each reference image in a preset image library; wherein the feature point cloud of the point cloud image is aligned with the feature point cloud of the first reference image after a set transformation process; The posture of the mobile device is controlled according to the first posture information reflected by the first reference image.

2. The method according to claim 1, characterized in that The step of obtaining a point cloud image of the pallet according to the color information and the depth information of the sampled image comprises: Inputting the sampled image into a preset segmentation model to extract a target area including the tray; A point cloud image of the pallet is obtained according to the target area and the set internal parameter calibration.

3. The method according to claim 1, characterized in that After obtaining a point cloud image of the pallet according to the color information and depth information of the sampled image, the method further includes: The point cloud image is subjected to data processing to obtain a processed point cloud image; wherein the data processing includes outlier denoising, downsampling, and removal of point cloud planes whose heights are lower than a set height threshold.

4. The method according to claim 1, characterized in that: Before determining the first reference image from among the reference images in the preset image library, the method further includes: Determining an offset angle of the pallet relative to the mobile device according to the outline of the pallet reflected by the point cloud image; The step of determining a first reference image from among the reference images in the preset image library comprises: Selecting, from each reference image in a preset image library, a second reference image whose offset angle is within a set angle threshold; Registering the second reference image and the point cloud image to obtain a registration result; The registration result is determined to be a second reference image that meets a set convergence condition and is used as the first reference image.

5. The method according to claim 4, characterized in that The registering the second reference image and the point cloud image to obtain a registration result includes: According to the set rigid body transformation parameters, the point cloud image is subjected to rigid body transformation processing; The minimized residual square sum between the second reference image after the rigid body transformation processing and the point cloud image is determined and used as the registration result.

6. The method according to claim 1, characterized in that Before controlling the posture of the mobile device according to the first posture information reflected by the first reference image, the method further includes: Determining rigid body transformation parameters corresponding to the first reference image; The controlling the posture of the mobile device according to the first posture information reflected by the first reference image includes: Obtaining second posture information of the mobile device relative to the tray in a camera coordinate system according to the first posture information reflected by the first reference image and a rigid body transformation parameter corresponding to the first reference image; According to the set conversion relationship and the second posture information, third posture information of the mobile device relative to the tray in the vehicle coordinate system is obtained; wherein the conversion relationship reflects that different points in the camera coordinate system correspond to different points in the vehicle coordinate system; The posture of the mobile device is controlled according to the third posture information.

7. A docking control device for a mobile device, characterized in that: include: An acquisition module, used for acquiring a sample image of the tray through an image acquisition device of a mobile device; An obtaining module, used for obtaining a point cloud image of the pallet according to the color information and depth information of the sampled image; A determination module, configured to determine a first reference image similar to the point cloud image from among the reference images in a preset image library; A control module is used to control the posture of the mobile device according to the first posture information reflected by the first reference image.

8. An electronic device, characterized in that: The method comprises a processor connected to a memory, wherein the memory is used to store a computer program; the processor is used to execute the computer program to implement the method according to any one of claims 1 to 6.

9. A mobile device, characterized in that: The docking control device of the mobile device as claimed in claim 7 or the electronic device as claimed in claim 8.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer-readable program, and the computer-readable program is executed by a processor to perform the steps of the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Equipment positioning method, device and system based on point cloud matching and medium

    CN114723970A

  • Tray pose identification method and device, electronic equipment and readable storage medium

    CN117523516A

  • Combined docking guiding method for split type hovercars

    CN117706595A

  • Target positioning method and system based on stereoscopic vision point cloud

    CN117994329A