Method, device and system for assisting docking, docking method and engineering equipment

Abstract

The present invention relates to the field of engineering equipment, and discloses a method, device and system for assisting docking, a docking method and an engineering equipment. The method for assisting docking includes: acquiring images of a first object and a second object to be docked; based on the images, determining a first central axis of the first object along the docking direction and a second central axis of the second object along the docking direction; determining a tangential line perpendicular to the docking direction of any one of the first object and the second object; and marking the first central axis, the second central axis and the tangential line in the images to assist the docking of the first object and the second object. Thereby, it is achieved that when assisting docking, the current postures of the first object and the second object are considered, and the marked first central axis, second central axis and tangential line can adapt to different postures of the first object and the second object.

Description

Technical Field

[0001] The present invention relates to the field of engineering equipment, and specifically, to a method, device and system for assisting docking, a docking method and an engineering equipment. Background Art

[0002] In the field of engineering equipment, before a vehicle-mounted pumping device conveys materials to a target position area, it is necessary to first dock the vehicle-mounted material pipeline with other conveying pipelines. Since the pipelines for conveying materials are rigid structures, the pipe orifices of the two pipelines must be aligned to carry out the next material conveyance. At the present stage, the docking of pipelines still uses traditional rear-view mirrors or requires the cooperation of other commanders to complete the pipeline docking, and the docking process is time-consuming.

[0003] In order to improve the docking efficiency of the pipelines of material equipment, a method for assisting docking has been proposed in the prior art. This method loads a pre-known calibration mark according to the position of the equipment pipeline in the image and the steering information of the vehicle, so as to assist the operator to complete the pipe alignment operation. However, the relevant method when the tapered pipe is in any attitude is not mentioned. Figure 1 、 Figure 2 、 Figure 3 Are the main diagrams in the method for assisting docking. Among them, Figure 1 Is a schematic diagram of the docking of the tapered pipe of the vehicle-mounted pump and the conveying pipe. In actual engineering operations, the tapered pipe and the conveying pipe need to be aligned to complete the material conveyance; Figure 2 Is a schematic diagram of the auxiliary line when the tapered pipe and the conveying pipe are parallel, showing the auxiliary docking line with the orifice of the tapered pipe as the base point when the radial directions of the tapered pipe and the conveying pipe are parallel; Figure 3 Is a schematic diagram of the auxiliary line when the tapered pipe and the conveying pipe are not parallel. When the conveying pipe and the tapered pipe are not radially parallel, the distal end of the auxiliary docking line displayed on the display screen marks the current steering information of the vehicle. As the vehicle attitude is adjusted, the steering auxiliary line is corrected synchronously. After the camera for assisting in pipe alignment is fixedly installed on the vehicle, the entire set of equipment needs to be calibrated. The pre-known calibration mark information loaded in the method for assisting docking is calibrated based on the condition that the attitude of the tapered pipe and the vehicle remain unchanged. However, in actual engineering operations, the tapered pipe needs to be adjusted up and down accordingly according to the inclination attitude of the conveying pipe. As Figure 4 Can be seen, for different rotation postures of the tapered pipe, the position of the orifice surface is different, which will cause differences in the calibration information on the image. Therefore, the calibration mark information mentioned in the method for assisting docking is difficult to adapt to different attitude situations of the tapered pipe. Summary of the Invention

[0004] The object of the present invention is to provide a method, device and system for assisting docking, a docking method and an engineering equipment, which can solve or at least partially solve the above problems.

[0005] To achieve the above object, an aspect of the present invention provides a method for assisting docking, the method comprising: acquiring images of a first object and a second object to be docked; based on the images, determining a first central axis of the first object along the docking direction and a second central axis of the second object along the docking direction; determining a tangential line perpendicular to the docking direction of any one of the first object and the second object; and annotating the first central axis, the second central axis and the tangential line in the images to assist the docking of the first object and the second object.

[0006] Optionally, determining the first central axis and / or the second central axis based on the images includes: determining a first coverage area of the first object in the images and / or a second coverage area of the second object in the images; determining a first skeleton line of the first coverage area and / or a second skeleton line of the second coverage area; and using a curve fitting method to fit the first skeleton line into the first central axis and / or fit the second skeleton line into the second central axis.

[0007] Optionally, determining the first coverage area of the first object in the images and / or the second coverage area of the second object in the images includes: determining the first coverage area and / or the second coverage area based on a preset convolutional neural network.

[0008] Optionally, determining the tangential line perpendicular to the docking direction of any one of the first object and the second object includes: determining the slope of the central axis of the any one; and based on the determined slope of the central axis and a preset look-up table corresponding to the any one, determining the tangential line of the any one, wherein the preset look-up table includes a correspondence between the slope of the central axis and the relevant information of the tangential line.

[0009] Optionally, the tangential line is located at a fixed position of any one of them. The information related to the tangential line includes the first coordinate of the first point on the tangential line and the second coordinate of the second point. The distance from the first point to the fixed position is a first preset distance, and the distance from the second point to the fixed position is a second preset distance. Determining the tangential line of any one of them based on the determined slope of the central axis and the preset lookup table corresponding to any one of them includes: when the determined slope of the central axis exists in the preset lookup table, finding the first coordinate and the second coordinate corresponding to the determined slope of the central axis in the preset lookup table; and determining the tangential line of any one of them according to the found first coordinate and second coordinate; and / or when the determined slope of the central axis does not exist in the preset lookup table, finding the first approximate central axis slope and the second approximate central axis slope closest to the determined slope of the central axis in the preset lookup table; finding the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope in the preset lookup table; determining the first coordinate and the second coordinate corresponding to the determined slope of the central axis based on the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope; and determining the tangential line of any one of them based on the first coordinate and the second coordinate corresponding to the determined slope of the central axis.

[0010] Optionally, the tangential line is located at the docking end of any one of them.

[0011] Optionally, any one of them is the one with less attitude change among the first object and the second object.

[0012] Correspondingly, another aspect of the present invention provides a device for assisting docking. The device includes: an image acquisition module for acquiring images of a first object and a second object to be docked; a central axis determination module for determining a first central axis of the first object along the docking direction and a second central axis of the second object along the docking direction based on the images; a tangential line determination module for determining a tangential line perpendicular to the docking direction of any one of the first object and the second object; and a marking module for marking the first central axis, the second central axis, and the tangential line in the images to assist the docking of the first object and the second object.

[0013] Optionally, the central axis determination module determining the first central axis and / or the second central axis based on the image includes: determining a first coverage area of the first object in the image and / or a second coverage area of the second object in the image; determining a first skeleton line of the first coverage area and / or a second skeleton line of the second coverage area; and using a curve fitting method to fit the first skeleton line into the first central axis and / or fit the second skeleton line into the second central axis.

[0014] Optionally, determining the first coverage area of the first object in the image and / or the second coverage area of the second object in the image includes: determining the first coverage area and / or the second coverage area based on a preset convolutional neural network.

[0015] Optionally, the tangential line determination module determining a tangential line perpendicular to the docking direction of any one of the first object and the second object includes: determining a central axis slope of the any one; and determining the tangential line of the any one based on the determined central axis slope and a preset look-up table corresponding to the any one, where the preset look-up table includes a correspondence between the central axis slope and tangential line related information.

[0016] Optionally, the tangential line is located at a fixed position of the any one, the tangential line related information includes a first coordinate of a first point on the tangential line and a second coordinate of a second point, the distance from the first point to the fixed position is a first preset distance, and the distance from the second point to the fixed position is a second preset distance. Determining the tangential line of the any one based on the determined central axis slope and the preset look-up table corresponding to the any one includes: when the determined central axis slope exists in the preset look-up table, finding the first coordinate and the second coordinate corresponding to the determined central axis slope in the preset look-up table; and determining the tangential line of the any one according to the found first coordinate and second coordinate; and / or when the determined central axis slope does not exist in the preset look-up table, finding a first approximate central axis slope and a second approximate central axis slope closest to the determined central axis slope in the preset look-up table; finding the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope in the preset look-up table; determining the first coordinate and the second coordinate corresponding to the determined central axis slope based on the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope; and determining the tangential line of the any one based on the first coordinate and the second coordinate corresponding to the determined central axis slope.

[0017] Optionally, the tangential line is located at the docking end of any one of them.

[0018] Optionally, any one of them is the one with less attitude change among the first object and the second object.

[0019] In addition, another aspect of the present invention further provides a docking method, which includes: controlling the docking of the first object and the second object according to the first central axis, the second central axis and the tangential line marked by the above-mentioned method for assisting docking.

[0020] In addition, another aspect of the present invention further provides a system for assisting docking, which includes: the above-mentioned device for assisting docking; and a display module for displaying an image marked with the first central axis, the second central axis and the tangential line.

[0021] Optionally, the system further includes: a camera module for acquiring videos of the first object and the second object; and a decomposition module for decomposing the acquired videos into images.

[0022] In addition, another aspect of the present invention further provides an engineering equipment, which includes the above-mentioned system for assisting docking.

[0023] In addition, another aspect of the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to cause a machine to execute the above-mentioned method for assisting docking or the docking method.

[0024] Through the above technical solutions, the first central axis, the second central axis and the tangential line are determined based on the acquired image, and the determined first central axis, second central axis and tangential line are marked in the image for assisting docking. The first central axis, second central axis and tangential line marked in the image are not pre-set, but are based on the real-time acquired image. In this way, the current postures of the first object and the second object are considered during the auxiliary docking; the first central axis, second central axis and tangential line for assisting docking are marked according to the current postures of the first object and the second object, and when the posture changes, the marked first central axis, second central axis and tangential line will also change. In this way, the marked first central axis, second central axis and tangential line can adapt to different postures of the first object and the second object. In addition, considering the postures of the first object and the second object during the auxiliary docking can make the docking process more accurate and the effect of the auxiliary docking better.

[0025] Other features and advantages of the present invention will be described in detail in the subsequent specific implementation part. Description of the Drawings

[0026] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation to the present invention. In the accompanying drawings:

[0027] Figure 1 is a schematic diagram of the butt joint between the conical pipe and the delivery pipe of a vehicle-mounted pump in the prior art;

[0028] Figure 2 is a schematic diagram of the auxiliary line when the conical pipe and the delivery pipe are parallel in the prior art;

[0029] Figure 3 is a schematic diagram of the auxiliary line when the conical pipe and the delivery pipe are not parallel in the prior art;

[0030] Figure 4 is a schematic diagram of different postures of the conical pipe;

[0031] Figure 5 is a flowchart of a method for assisting butt joint provided by an embodiment of the present invention;

[0032] Figure 6 is a schematic diagram of the central axis provided by another embodiment of the present invention;

[0033] Figure 7 is a schematic diagram of the butt joint when there is a radial deviation between the conical pipe and the delivery pipe provided by another embodiment of the present invention;

[0034] Figure 8 is a schematic diagram of two pipes to be butt jointed provided by another embodiment of the present invention;

[0035] Figure 9 is provided by another embodiment of the present invention Figure 8 a schematic diagram of the covered area of the two pipes shown;

[0036] Figure 10 is a schematic diagram of the equipment installation in the auxiliary butt joint provided by another embodiment of the present invention;

[0037] Figure 11 is a schematic diagram of the central axis and the tangential line of the pipe orifice provided by another embodiment of the present invention;

[0038] Figure 12 is a top view of the vertebral canal after installing the calibration tooling provided by another embodiment of the present invention;

[0039] Figure 13 is a front view of the vertebral canal after installing the calibration tooling provided by another embodiment of the present invention;

[0040] Figure 14 is a schematic diagram of the vertebral canal installation calibration tooling provided by another embodiment of the present invention;

[0041] Figure 15 It is a schematic diagram of the spinal canal in different postures provided by another embodiment of the present invention;

[0042] Figure 16 It is a flowchart of a method for assisting docking provided by another embodiment of the present invention; and

[0043] Figure 17 It is a structural block diagram of a device for assisting docking provided by another embodiment of the present invention.

[0044] Description of reference numerals

[0045] 1 Delivery pipe 2 Taper pipe

[0046] 3 Camera 4 Display

[0047] 5 Vehicle body 6 Image acquisition module

[0048] 7 Central axis determination module 8 Tangential line determination module

[0049] 9 Marking module Detailed implementation manners

[0050] The following details the specific implementation manners of the present invention with reference to the accompanying drawings. It should be understood that the specific implementation manners described herein are only for explaining and understanding the present invention, and are not used to limit the present invention.

[0051] One aspect of the embodiments of the present invention provides a method for assisting docking.

[0052] Figure 5 It is a flowchart of a method for assisting docking provided by an embodiment of the present invention. As Figure 5 shown, the method includes the following.

[0053] In step S50, images of the first object and the second object to be docked are acquired. Specifically, the images of the first object and the second object can be directly taken, for example, using a camera for taking; or it can be taking videos of the first object and the second object, for example, using a camera for taking, and performing frame-by-frame processing on the taken videos to obtain the images of the first object and the second object. No matter what method is used, as long as the images of the first object and the second object can be acquired.

[0054] In step S51, based on the images, a first central axis of the first object along the docking direction and a second central axis of the second object along the docking direction are determined. Among them, the docking direction is the direction along which the first object and the second object are docked together. As Figure 6 shown, what are to be docked are the delivery pipe 1 and the taper pipe 2 of a vehicle-mounted pump, and the two pipes are docked horizontally along the transverse direction of the pipes, the docking direction is the horizontal direction, and the central axes of the two pipes are as Figure 6as shown by the dashed line in

[0055] In step S52, a tangential line perpendicular to the docking direction of either the first object or the second object is determined. The tangential line can help determine the spacing between the first object and the second object in the direction perpendicular to the docking direction. As Figure 7 shown, the objects to be docked are the delivery pipe 1 of the vehicle-mounted pump and the tapered pipe 2. The two pipes are docked along the transverse direction of the pipes. The docking direction is the transverse direction, and the direction perpendicular to the docking direction is the radial direction. The tangential line helps determine the spacing between the delivery pipe 1 and the tapered pipe 2 in the radial direction. By helping determine the spacing between the first object and the second object in the direction perpendicular to the docking direction, docking can be better assisted. For example, the slope of the central axis of either one is determined, where the slope of the central axis is the slope of the straight line where the central axis of either one is located. Specifically, a coordinate system can be established, and any two points on the central axis are selected, and the slope of the central axis is determined based on the selected two points. Based on the determined slope of the central axis and the corresponding preset lookup table of either one, the tangential line of either one is determined, where the preset lookup table includes the correspondence between the slope of the central axis and the relevant information of the tangential line. In addition, the first object and the second object can be distinguished based on the characteristics of the first object and the second object. For example, during the docking process of the first object and the second object, the first object is stationary relative to the imaging device, and the second object moves. By comparing two adjacent pictures, the one without change is the first object, and the one with change is the second object. Specifically, whether the first object and the second object have changes can be compared by distinguishing the covered areas of the first object and the second object in the image respectively. After distinguishing the first object and the second object, the tangential line of either one is determined according to the corresponding preset lookup table of the selected one. In addition, the relevant information of the tangential line is used to determine the tangential line. For example, the relevant information of the tangential line can include the coordinates of any two points, or can include the slope and the coordinates of one point, and so on. Specifically, the relevant information of the tangential line corresponding to the determined slope of the central axis is determined based on the preset lookup table, and the tangential line of either one is determined according to the determined tangential line information. In addition, the method of establishing the coordinate system when determining the slope of the central axis of either one is the same as the method of establishing the coordinate system when the slope of the central axis in the preset lookup table is determined.

[0056] In step S53, the first central axis, the second central axis, and the tangential line are marked in the image to assist the docking of the first object and the second object.

[0057] Through the above technical solution, the first central axis, the second central axis, and the tangential line are determined based on the acquired image, and the determined first central axis, second central axis, and tangential line are marked on the image for assisting docking. The first central axis, second central axis, and tangential line marked on the image are not pre-set, but are based on the real-time acquired image. In this way, the current postures of the first object and the second object are considered during the assisting docking. The first central axis, second central axis, and tangential line for assisting docking are marked according to the current postures of the first object and the second object. When the posture changes, the marked first central axis, second central axis, and tangential line will also change. In this way, the marked first central axis, second central axis, and tangential line can adapt to the different postures of the first object and the second object. In addition, considering the postures of the first object and the second object during the assisting docking can make the docking process more accurate and the effect of the assisting docking better.

[0058] There are many ways to determine the first central axis of the first object and / or the second central axis of the second object. Optionally, in the embodiments of the present invention, the first central axis and / or the second central axis can be determined according to the following content. Determine the first coverage area of the first object in the image and / or the second coverage area of the second object in the image. As Figure 8 shown, the first object and the second object are two pipes, the conveying pipe 1 and the tapered pipe 2; the coverage areas of the two pipes in the image are determined respectively, as Figure 9 shown by the white parts in the figure, which are the coverage area of the conveying pipe 1 and the coverage area of the tapered pipe 2 respectively. Among them, in the embodiments of the present invention, a preset convolutional neural network can be used to determine the first coverage area and / or the second coverage area. The preset convolutional neural network is a trained convolutional neural network. Inputting the images of the first object and the second object into the preset convolutional neural network can obtain the first coverage area and / or the second coverage area. In addition, an image processing algorithm can also be used to determine the first coverage area and / or the second coverage area. For example, computer vision algorithms such as threshold segmentation method, watershed algorithm, edge extraction, and morphological operation are combined. Determine the first skeleton line of the first coverage area and / or the second skeleton line of the second coverage area. Optionally, in the embodiments of the present invention, there are many ways to determine the skeleton line. For example, the skeleton line is calculated by combining the distance transformation method and the threshold segmentation method; or, the zhang-suen thinning algorithm can also be used to extract the skeleton line. Using the curve fitting method, the first skeleton line is fitted into the first central axis and / or the second skeleton line is fitted into the second central axis. In addition, in the embodiments of the present invention, the image including the coverage area can be a binary image or an image of other colors, as long as the coverage area can be distinguished from the non-coverage area.

[0059] In addition, in the embodiments of the present invention, a convolutional neural network can be trained according to the following content to obtain a preset convolutional neural network. 1) Record video data of the docking process of the first object and the second object. For example, record it using a camera. For example, when the conical pipe and the delivery pipe of a vehicle-mounted pump are to be docked, record the video data of the pipe alignment process of the conical pipe and the delivery pipe, and use a camera for recording, as Figure 10 shown. The camera 3 is installed on the vehicle body 5 of the vehicle-mounted pump to record the docking process of the delivery pipe 1 and the conical pipe 2. In addition, when recording the video data of the docking process of the conical pipe and the delivery pipe, different working conditions can be fully considered, and the video can be recorded under different working conditions to enrich the data for training the convolutional neural network, making the processing of the preset neural network more accurate and improving the accuracy of the method for assisting docking provided by the embodiments of the present invention. 2) Data screening. First, frame the video recorded in 1) to obtain images of the first object and the second object. In addition, in order to make the images obtained by framing more effective for training the convolutional neural network and improve the training speed, effective images can be selected from the framed images for training the convolutional neural network. Specifically, the principle for selecting effective images is that the selected images are different, and the positions and angles of the first object and / or the second object in the selected images are different. Finally, collect a preset number of images including the first object and the second object. For example, collect 10,000 images. The larger the number of collected images, the better the training effect of the convolutional neural network. 3) Data annotation. Perform pixel-level semantic annotation on the screened images of the first object and the second object. Among them, in the embodiments of the present invention, performing pixel-level semantic annotation is to determine the coverage areas of the first object and the second object. For example, when docking the conical pipe and the delivery pipe of a vehicle-mounted pump, as Figure 8 and Figure 9 shown, from Figures 8 to 9 this is pixel-level semantic annotation, that is, to determine the coverage areas of the delivery pipe 1 and the conical pipe 2. 4) Model training. Divide the annotated data set into a training set and a test set, and send the training set images into a semantic segmentation model based on a convolutional neural network for training. Among them, the annotated data set includes the original images of the first object and the second object and the annotated images corresponding to the original images, where the first object and the second object's coverage areas have been determined. One original image corresponds to its annotated image. For example, taking the docking of the delivery pipe and the conical pipe of a vehicle-mounted pump as an example, the annotated data set includes the original images of the delivery pipe and the conical pipe (as Figure 8 shown) and the annotated images determining the coverage areas of the delivery pipe and the conical pipe (as Figure 9 shown). The test set is used to test the trained convolutional neural network. After the test is completed, a trained convolutional neural network is obtained.

[0060] Optionally, in the embodiments of the present invention, the tangential line is located at a fixed position of any one. The tangential line related information includes the first coordinate of the first point and the second coordinate of the second point located on the tangential line. The distance from the first point to the fixed position is the first preset distance, and the distance from the second point to the fixed position is the second preset distance. Determining the tangential line of any one based on the determined central axis slope and the preset look-up table may include the following. Compare the central axis slope of any one of the determined first object and the second object with the central axis slopes included in the preset look-up table to determine whether the central axis slope of any one of the determined ones is in the preset look-up table. When the central axis slope of any one of the determined ones is in the preset look-up table, find the first coordinate and the second coordinate corresponding to the central axis slope of any one of the determined ones in the preset look-up table, and then determine the tangential line of any one based on the found first coordinate and the second coordinate. And / or, when the central axis slope of any one of the determined ones is not in the preset look-up table, find the first approximate central axis slope and the second approximate central axis slope closest to the central axis slope of any one of the determined ones in the preset look-up table. Find the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope in the preset look-up table. Based on the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope, determine the first coordinate and the second coordinate corresponding to the central axis slope of any one of the determined ones. Specifically, the interpolation method can be used to determine the first coordinate and the second coordinate corresponding to the central axis slope of any one of the determined ones. For example, determine the midpoint coordinate of the first coordinate corresponding to the first central axis slope and the first coordinate corresponding to the second central axis slope as the first coordinate corresponding to the central axis slope of any one of the determined ones by the interpolation method, and determine the midpoint coordinate of the second coordinate corresponding to the first central axis slope and the second coordinate corresponding to the second central axis slope as the second coordinate corresponding to the central axis slope of any one of the determined ones by the interpolation method. In this way, the first coordinate and the second coordinate corresponding to the central axis slope of any one of the determined ones are determined. Based on the first coordinate and the second coordinate corresponding to the central axis slope of any one of the determined ones, determine the tangential line of any one.

[0061] Optionally, in the embodiments of the present invention, the tangential line is located at the docking end of any one. Wherein, the docking end is the end of any one that contacts the other when docking with the other of the first object and the second object. As Figure 10 shown, the ones being docked are the delivery pipe 1 and the tapered pipe 2, and any one is the tapered pipe 2. When the delivery pipe 1 and the tapered pipe 2 are docked, the A end of the tapered pipe 2 contacts the delivery pipe 1, and the A end is the docking end. When the tangential line is located at the docking end, the distance between the docking end of any one and the docking end of the other of the first object and the second object can be more intuitively marked, which is convenient for controlling the docking of the first object and the second object. AsFigure 10 As shown, the tangential line is located at the A end of the tapered pipe 2, which can visually indicate the distance between the A end of the tapered pipe 2 and the B end of the conveying pipe 1, where the B end is the docking end of the conveying pipe 1.

[0062] Optionally, in the embodiment of the present invention, either one is the one with less attitude change among the first object and the second object. For example, when the conveying pipe of the vehicle-mounted pump and the tapered pipe are to be docked, the attitude change of the tapered pipe is less, and either one can be the tapered pipe. The tangential line can reflect the attitude of either one, and docking is performed according to the attitude of either one during docking. When the tangential line is the one with less attitude change among the first object and the second object, docking can be achieved as soon as possible with less attitude change. In addition, if either one is the one with less attitude change among the first object and the second object, then the amount of data in the preset lookup table including the corresponding relationship between the slope of the central axis of either one and the first coordinate and the second coordinate will be less. With less data volume, it is convenient for data processing and the processing speed can be improved.

[0063] Optionally, in the embodiment of the present invention, a preset lookup table can be established according to the following content. Among them, the first object corresponds to a preset lookup table, and the second object corresponds to a preset lookup table. Hereinafter, an example of establishing the preset lookup table corresponding to the first object will be described, and the establishment of the preset lookup table corresponding to the second object can be carried out with reference to the following content. Install a calibration tooling part on the first object. The calibration tooling part is used to represent the tangential line, and the tooling part includes a straight rod part, and the straight rod part marks the tangential line. For example, the first object is a tapered pipe, and the tangential line is located at the docking end of the tapered pipe, as Figure 10 shown. In addition, as Figure 10 shown, during actual use, one end of the tapered pipe is connected to the vehicle body 5, and the other end is docked with the conveying pipe 1. The A end is a free pipe orifice. Therefore, it can be said that the tangential line is located at the pipe orifice of the tapered pipe 2, and the tangential line of the tapered pipe is the pipe orifice tangential line, as Figure 11 shown. No matter how the tooling part is installed, as long as the tooling part can be fixed at a fixed position on the first object. For example, taking the first object and the second object being the tapered pipe and the conveying pipe respectively, and the tangential line being located at the pipe orifice of the tapered pipe as an example, the calibration tooling part can be designed to consist of a "T"-shaped rigid structural part and two anchor points. The "T"-shaped rigid structural part is composed of two connecting rods perpendicular to each other, namely a long connecting rod and a short connecting rod. The diameter of the short connecting rod is the same as the inner diameter of the pipe orifice of the tapered pipe. The two anchor points are respectively arranged at both ends of the long connecting rod. The long connecting rod is the straight rod part used to represent the tangential line in the embodiment of the present invention, and the short connecting rod is used to fix the calibration tooling part. The two ends of the long connecting rod are the two anchor points. Insert the short connecting rod of the calibration tooling part into the pipe orifice of the tapered pipe, as Figure 14 shown, rotate the long connecting rod with the short connecting rod as the axis to ensure that the long connecting rod is parallel to the ground, so as to complete the installation of the calibration tooling part. The installation effect diagram of the calibration tooling part can be referred toFigure 12 and Figure 13As shown. The short connecting rod part of the calibration tooling part can also be designed into other structures, as long as the calibration tooling part can be fixed. A point on the straight rod part of the calibration tooling part at a first preset distance from the fixed position where the tangential line is located is the first point, and a point on the straight rod part at a second preset distance from the fixed position where the tangential line is located is the second point. The line connecting the first point and the second point is the tangential line. For example, the two anchor points on the above-mentioned "T"-shaped rigid structure part are the first point and the second point respectively, and connecting the two anchor points can obtain the tangential line of the pipe orifice. Record the video of the first object and adjust the posture of the first object. Among them, each time the posture of the first object is adjusted, record the slope of the central axis of the first object and the first coordinate of the first point and the second coordinate of the second point representing the current tangential line of the first object, that is, each time the posture of the first object changes, record the slope of the central axis of the first object and the first coordinate and the second coordinate. In this way, a preset lookup table corresponding to the first object is obtained. The obtained preset lookup table records the corresponding relationship between the slope of the central axis of the first object and the first coordinate and the second coordinate. When obtaining the slope of the central axis and the first coordinate and the second coordinate, it can be the image obtained by frame-dividing the recorded video. The method for obtaining the slope of the central axis can refer to the method described in the above-mentioned embodiment, determine the central axis of the first object, establish a coordinate system, select any two points on the central axis, and then determine the slope of the central axis according to the selected two points. In addition, there are many methods for obtaining the first coordinate and the second coordinate. For example, after obtaining the image of the first object, a coordinate system can be established, and the first coordinate of the first point and the second coordinate of the second point can be set. In addition, after obtaining the image of the first object, the first point area and the second point area can be extracted, a coordinate system can be established, and the coordinates of the center of gravity or centroid of the first point area can be calculated to obtain the first coordinate and the coordinates of the center of gravity or centroid of the second point area can be calculated to obtain the second coordinate. The first point area refers to the area occupied by the first point, and the second point area refers to the area occupied by the second point. For example, taking the tangential line of the pipe orifice of the above-mentioned tapered pipe as an example, the two anchor points are the first point and the second point respectively. After obtaining the image of the tapered pipe, a coordinate system is established, and the coordinates of the anchor points are set; it can also be that the anchor point area is set to a dark color, for example, red, the dark area is extracted, a coordinate system is established, and the coordinates of the center of gravity or centroid of the dark area are obtained, and the coordinates of the anchor points can be obtained. In addition, for subsequent precise searching, the dense sampling method is used to collect as many slope values of the central axis and the corresponding first coordinates and second coordinates as possible. In addition, in the embodiment of the present invention, in addition to using the calibration tooling part to represent the tangential line, a laser beam can also be used to represent the tangential line. For example, taking the tangential line located on the first object as an example, the laser beam is located at the fixed position of the first object. A point on the laser beam at a first preset distance from the fixed position is the first point, and a point on the laser beam at a second preset distance from the fixed position is the second point. After obtaining the image of the first object, the first coordinate and the second coordinate can be determined by referring to the method described in the above-mentioned embodiment.

[0064] In addition, taking the conveying pipe, the tapered pipe, and the tangential line to be docked being located at the pipe orifice of the tapered pipe as an example, when determining the tangential line of the pipe orifice of the tapered pipe using the preset lookup table established based on the "T"-shaped calibration tooling part, it is similar to the method for determining the tangential line of any one of them in the above embodiments. Specifically, refer to the following content. Determine the slope of the central axis of the tapered pipe, and check whether the determined slope of the central axis of the tapered pipe exists in the preset lookup table corresponding to the tapered pipe. If the determined slope of the central axis of the tapered pipe exists in the preset lookup table, directly find the first coordinate and the second coordinate corresponding to the determined slope of the central axis in the preset lookup table, that is, find the coordinates corresponding to two anchor points, and based on the found coordinates, determine the tangential line of the pipe orifice. If the determined slope of the central axis of the tapered pipe does not exist in the preset lookup table, find the first approximate central axis slope and the second approximate central axis slope closest to the determined slope of the central axis of the tapered pipe in the preset lookup table. For example, they are K1 and K2 respectively. Determine the coordinates of the two anchor points corresponding to K1. For example, they are P 11 and P 12 , determine the coordinates of the two anchor points corresponding to K2. For example, they are P 21 and P 22 , where P 11 and P 21 are different coordinates of the same anchor point, and P 12 and P 22 are different coordinates of another anchor point. Calculate the coordinates P1 of the midpoint of P 11 and P 21 by interpolation method, and calculate the coordinates P2 of the midpoint of P 12 and P 22 . The line connecting the points corresponding to the coordinates P1 and P2 respectively is the tangential line of the pipe orifice of the current tapered pipe attitude.

[0065] Figure 16 is a flowchart of a method for assisting docking provided by another embodiment of the present invention. As Figure 16 shown, in this embodiment, the method includes the following content. Among them, in this embodiment, the tapered pipe of the vehicle-mounted pump and the conveying pipe are to be docked, and the tangential line is located at the pipe orifice of the tapered pipe. The tangential line of the pipe orifice of the tapered pipe is calibrated by a "T"-shaped calibration tooling part.

[0066] Input the target pipe image, where the target pipe includes a tapered pipe and a conveying pipe. Call the semantic segmentation model to predict the target pipe region, that is, determine the coverage region of the tapered pipe in the image and the coverage region of the conveying pipe in the image. The semantic segmentation model is a model based on a convolutional neural network and is pre-trained. Identify the tapered pipe and the conveying pipe regions. For example, by comparing two adjacent input images, the region with unchanged coverage is the tapered pipe because the tapered pipe is installed on the vehicle body, as Figure 10As shown, the camera and the tapered tube are relatively stationary, and the covered area of the tapered tube in the image does not change. Calculate the central axes of the tapered tube and the conveying tube. Calculate the slope of the central axis of the tapered tube, and calculate the tangential line of the tube orifice of the tapered tube according to the slope of the central axis of the tapered tube. Input the auxiliary line of the target tube to the display screen, where the auxiliary line of the target tube includes the central axis of the tapered tube, the central axis of the conveying tube, and the tangential line of the tube orifice of the tapered tube. In the technical solution provided by the embodiment of the present invention, the tube alignment assisting device may include a vision sensor, an intelligent host, and a display screen, where the vision sensor may be a camera. Among them, the installation schematic diagram of the tube alignment device is as Figure 10 shown. The purpose of assisting tube alignment is to collect the image videos of the tapered tube and the conveying tube through the vision sensor, and then identify the positions and their positional relationships of the tapered tube and the conveying tube through the AI algorithm of the intelligent host. Finally, the central axes of the tapered tube and the conveying tube and the tangential line of the tube orifice of the tapered tube are displayed on the display screen as auxiliary lines to guide the driver to adjust the vehicle attitude. Figure 11 is the schematic diagram of tube alignment collected from the perspective of the camera. According to the pinhole imaging model of the camera, it can be known that if the central axes of the tapered tube and the conveying tube are parallel on the image coordinate, it indicates that the central axes of the two target tubes are also parallel in the world coordinate. From Figure 7It can be seen that if there is a large deviation in the front-back position (radial direction) between the tapered pipe and the conveying pipe, it is very difficult to judge the radial distance between the tapered pipe and the conveying pipe only through the positional relationship between the two pipes in the video image without the assistance of the tangential line of the pipe orifice of the tapered pipe. Therefore, the tangential line of the tapered pipe orifice helps the driver accurately judge the radial deviation between the tapered pipe and the conveying pipe in the video image, so as to purposefully adjust the vehicle attitude. In short, during the process of assisting in pipe alignment, usually the conveying pipe is in a static state in the world coordinate system. The driver can adjust the vehicle attitude to complete the pipe alignment operation according to the parallel relationship between the central axes of the tapered pipe and the conveying pipe and the positional relationship between the orifice of the conveying pipe and the tangential line of the orifice of the tapered pipe in the video image. The calculation method of the central axes of the tapered pipe and the conveying pipe is realized based on machine vision technology: (1) First, train a semantic segmentation network model; (2) Then, infer the target pipe area; (3) Then, calculate the area skeleton line according to the technical area of the target pipe area; (4) Finally, calculate the central axis of the target pipe by means of curve fitting according to the skeleton line of the target pipe area. In addition, when establishing the preset look-up table corresponding to the tapered pipe, the tangential line of the tapered pipe orifice is realized by means of calibration: (1) First, design a set of calibration tooling; (2) Calibrate the tangential line of the tapered pipe orifice through the calibration tooling; (3) Adjust the attitude of the tapered pipe, and collect the information of the tangential line of the pipe orifice under different attitudes of the tapered pipe by means of dense sampling. In actual pipe alignment operations, the attitude of the tapered pipe is not fixed. The calibration information obtained based on a certain attitude of the tapered pipe cannot well adapt to other attitudes of the tapered pipe. The technical solution provided in the embodiment of the present invention proposes a design of a calibration tooling and a method for calibrating the tangential line of the pipe orifice under different attitudes of the tapered pipe, which can solve the problem of calculating the calibration information under various attitudes of the tapered pipe. In the prior art, calibration is performed once and used multiple times. No matter what the tapered pipe looks like, the same calibration information is used. The auxiliary alignment line generated by the method provided in the technical solution of the embodiment of the present invention is dynamically adjusted in real time according to the positions of the tapered pipe and the conveying pipe, and its flexibility and applicability to multiple working conditions are better than those of the prior method. In addition, the prior method does not mention how to obtain the attitude of the tapered pipe. In the technical solution provided in the embodiment of the present invention, the tangential line of the pipe orifice is used to characterize the attitude of the tapered pipe.

[0067] Correspondingly, another aspect of the embodiment of the present invention provides a device for assisting docking.

[0068] Figure 17 is a structural block diagram of a device for assisting docking provided in another embodiment of the present invention, as Figure 17As shown, the device includes an image acquisition module 6, a central axis determination module 7, a tangential line determination module 8, and a marking module 9. Among them, the image acquisition module 6 is used to acquire images of a first object and a second object to be docked; the central axis determination module 7 is used to determine a first central axis of the first object along the docking direction and a second central axis of the second object along the docking direction based on the images; the tangential line determination module 8 is used to determine a tangential line perpendicular to the docking direction of either the first object or the second object; and the marking module 9 is used to mark the first central axis, the second central axis, and the tangential line in the image to assist in the docking of the first object and the second object.

[0069] Optionally, in an embodiment of the present invention, the central axis determination module determines the first central axis and / or the second central axis based on the image, including: determining a first coverage area of the first object in the image and / or a second coverage area of the second object in the image; determining a first skeleton line of the first coverage area and / or a second skeleton line of the second coverage area; and using a curve fitting method to fit the first skeleton line into the first central axis and / or fit the second skeleton line into the second central axis.

[0070] Optionally, in an embodiment of the present invention, determining the first coverage area of the first object in the image and / or the second coverage area of the second object in the image includes: determining the first coverage area and / or the second coverage area based on a preset convolutional neural network.

[0071] Optionally, in an embodiment of the present invention, the tangential line determination module determines a tangential line perpendicular to the docking direction of either the first object or the second object, including: determining the slope of the central axis of either one; and determining the tangential line of either one based on the determined slope of the central axis and a preset look-up table corresponding to either one, where the preset look-up table includes the correspondence between the slope of the central axis and the relevant information of the tangential line.

[0072] Optionally, in an embodiment of the present invention, the tangential line is located at a fixed position of any one of them. The tangential line related information includes the first coordinate of the first point and the second coordinate of the second point located on the tangential line. The distance from the first point to the fixed position is a first preset distance, and the distance from the second point to the fixed position is a second preset distance. Determining the tangential line of any one of them based on the determined slope of the central axis and the corresponding preset lookup table includes: when the determined slope of the central axis exists in the preset lookup table, finding the first coordinate and the second coordinate corresponding to the determined slope of the central axis in the preset lookup table; and determining the tangential line of any one of them according to the found first coordinate and second coordinate; and / or when the determined slope of the central axis does not exist in the preset lookup table, finding the first approximate central axis slope and the second approximate central axis slope closest to the determined slope of the central axis in the preset lookup table; finding the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope in the preset lookup table; determining the first coordinate and the second coordinate corresponding to the determined slope of the central axis based on the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope; and determining the tangential line of any one of them based on the first coordinate and the second coordinate corresponding to the determined slope of the central axis.

[0073] Optionally, in an embodiment of the present invention, the tangential line is located at the docking end of any one of them.

[0074] Optionally, in an embodiment of the present invention, any one of them is the one with less attitude change among the first object and the second object.

[0075] The specific working principle and benefits of the device for assisting docking provided by the embodiment of the present invention are similar to those of the method for assisting docking provided by the embodiment of the present invention, and will not be elaborated here.

[0076] In addition, on the other hand, an embodiment of the present invention further provides a docking method, which includes: controlling the docking of the first object and the second object according to the first central axis, the second central axis and the tangential line marked by the method for assisting docking described in the above embodiment.

[0077] In addition, on the other hand, an embodiment of the present invention further provides a system for assisting docking, which includes: the device for assisting docking described in the above embodiment; and a display module for displaying an image marked with the first central axis, the second central axis and the tangential line.

[0078] Optionally, in an embodiment of the present invention, the system further includes: a camera module for acquiring videos of the first object and the second object; and a decomposition module for decomposing the acquired videos into images.

[0079] In addition, on the other hand, an embodiment of the present invention further provides a construction equipment, which includes the system for assisting docking described in the above embodiments.

[0080] In addition, on the other hand, an embodiment of the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to cause the machine to execute the method for assisting docking or the docking method described in the above embodiments.

[0081] The preferred embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solutions of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0082] In addition, it should be noted that, among the various specific technical features described in the above specific embodiments, they can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present invention will not separately describe various possible combination methods.

[0083] In addition, any combination can be made between various different embodiments of the present invention, as long as it does not violate the idea of the present invention, it should also be regarded as the content disclosed by the present invention.

Claims

1. A method for assisting docking, characterized in that, The method includes: obtaining images of a first object and a second object to be docked; based on the images, determining a first central axis of the first object along the docking direction and a second central axis of the second object along the docking direction; determining a tangential line of any one of the first object and the second object that is perpendicular to the docking direction and parallel to the ground; and marking the first central axis, the second central axis, and the tangential line in the images to assist in the docking of the first object and the second object, wherein determining the tangential line of any one of the first object and the second object that is perpendicular to the docking direction includes: determining the slope of the central axis of the any one; and based on the determined slope of the central axis and a preset look-up table corresponding to the any one, determining the tangential line of the any one, wherein the preset look-up table includes the correspondence between the slope of the central axis and the relevant information of the tangential line, and the preset look-up table is obtained by using the dense sampling method with a calibration tooling part composed of two mutually perpendicular connecting rods, wherein the tangential line is located at a fixed position of the any one, and the relevant information of the tangential line includes the first coordinate of a first point and the second coordinate of a second point on the tangential line, the distance from the first point to the fixed position is a first preset distance, and the distance from the second point to the fixed position is a second preset distance.

2. The method according to claim 1, wherein Determining the first central axis and / or the second central axis based on the images includes: determining a first coverage area of the first object in the images and / or a second coverage area of the second object in the images; determining a first skeleton line of the first coverage area and / or a second skeleton line of the second coverage area; and using a curve fitting method to fit the first skeleton line into the first central axis and / or fit the second skeleton line into the second central axis.

3. The method according to claim 2, wherein The determining the first coverage area of the first object in the images and / or the second coverage area of the second object in the images includes: determining the first coverage area and / or the second coverage area based on a preset convolutional neural network.

4. The method according to claim 1, wherein Determining the tangential line of any one based on the determined slope of the central axis and the preset look-up table corresponding to the any one includes: when the determined slope of the central axis exists in the preset look-up table, finding the first coordinate and the second coordinate corresponding to the determined slope of the central axis in the preset look-up table; and determining the tangential line of the any one according to the found first coordinate and second coordinate; and / or when the determined slope of the central axis does not exist in the preset look-up table, finding a first approximate central axis slope and a second approximate central axis slope that are closest to the determined slope of the central axis in the preset look-up table; finding the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope in the preset look-up table; Based on the first coordinates and the second coordinates corresponding to the first slope approaching the central axis and the first coordinates and the second coordinates corresponding to the second slope approaching the central axis, determine the first coordinates and the second coordinates corresponding to the determined slope of the central axis; and Based on the first coordinates and the second coordinates corresponding to the determined slope of the central axis, determine the tangential line of any one of them.

5. The method according to any one of claims 1-4, characterized in that, The tangential line is located at the docking end of any one of them.

6. The method according to any one of claims 1 to 4, characterized in that, Any one of them is the one with less attitude change among the first object and the second object.

7. A device for assisting docking, characterized in that, The device includes: An image acquisition module for acquiring images of a first object and a second object to be docked; A central axis determination module for determining a first central axis of the first object along the docking direction and a second central axis of the second object along the docking direction based on the images; A tangential line determination module for determining a tangential line perpendicular to the docking direction and parallel to the ground of any one of the first object and the second object; and A marking module for marking the first central axis, the second central axis and the tangential line in the image to assist in the docking of the first object and the second object, wherein, determining the tangential line perpendicular to the docking direction of any one of the first object and the second object includes: Determining the slope of the central axis of any one of them; and Based on the determined slope of the central axis and a preset lookup table corresponding to any one of them, determine the tangential line of any one of them, wherein the preset lookup table includes the correspondence between the slope of the central axis and the relevant information of the tangential line, and the preset lookup table is obtained by using the dense sampling method with a calibration tooling part composed of two mutually perpendicular connecting rods. Wherein, the tangential line is located at a fixed position of any one of them, and the relevant information of the tangential line includes the first coordinates of the first point on the tangential line and the second coordinates of the second point, the distance from the first point to the fixed position is a first preset distance, and the distance from the second point to the fixed position is a second preset distance.

8. The device according to claim 7, characterized in that, The central axis determination module determines the first central axis and / or the second central axis based on the images, including: Determining a first coverage area of the first object in the image and / or a second coverage area of the second object in the image; Determining a first skeleton line of the first coverage area and / or a second skeleton line of the second coverage area; and Using the curve fitting method, fitting the first skeleton line into the first central axis and / or fitting the second skeleton line into the second central axis.

9. The device according to claim 8, characterized in that, Determining the first coverage area of the first object in the image and / or the second coverage area of the second object in the image includes: Determining the first coverage area and / or the second coverage area based on a preset convolutional neural network.

10. The device according to claim 7, wherein Determining the tangential line of any one of them based on the determined slope of the central axis and a preset lookup table corresponding to any one of them includes: When the determined slope of the central axis is in the preset lookup table, Find the first coordinate and the second coordinate corresponding to the determined central axis slope in the preset look-up table; and Determine the tangent line of any one of them according to the found first coordinate and second coordinate; and / or In the case where the determined central axis slope is not in the preset look-up table,[ Find the first approximate central axis slope and the second approximate central axis slope closest to the determined central axis slope in the preset look-up table; Find the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope in the preset look-up table; Based on the first coordinate and the second coordinate corresponding to the first approximate central axis slope and the first coordinate and the second coordinate corresponding to the second approximate central axis slope, determine the first coordinate and the second coordinate corresponding to the determined central axis slope; and Based on the first coordinate and the second coordinate corresponding to the determined central axis slope, determine the tangent line of any one of them.

11. The device according to any one of claims 7 to 10, characterized in that, The tangent line is located at the docking end of any one of them.

12. The device according to any one of claims 7-10, characterized in that, Any one of them is the one with less attitude change among the first object and the second object.

13. A docking method, characterized in that, The docking method includes: Controlling the docking of the first object and the second object according to the first central axis, the second central axis and the tangent line marked by the method according to any one of claims 1-6.

14. A system for assisting docking, characterized in that, The system includes: The device according to any one of claims 7-10; and A display module for displaying an image marked with a first central axis, a second central axis and a tangent line.

15. The system according to claim 14, wherein The system further includes: A camera module for acquiring videos of the first object and the second object; and A decomposition module for decomposing the acquired videos into images.

16. An engineering equipment, characterized in that, The engineering equipment includes the system according to claim 14 or 15.

17. A machine-readable storage medium, characterized in that, Instructions are stored on the machine-readable storage medium, and the instructions are used to cause the machine to execute the method according to any one of claims 1-6 or 13.

Images