Automatic crane pipe alignment system and alignment method based on image analysis

Abstract

The invention relates to the field of crane pipe alignment based on image analysis, and discloses an automatic crane pipe alignment method based on image analysis, and the method comprises the steps: collecting a high-definition image, an infrared image and a radar image of a to-be-aligned crane pipe through a pre-configured multi-type image collection device; identifying a spectral region of the high-definition image, removing artifacts, and segmenting to obtain a preliminary alignment interface; identifying infrared pixel points of the infrared image, analyzing pixel offset level and edge information through a series of calculation, and calibrating the preliminary alignment interface to obtain a calibration interface; a three-dimensional coordinate graph is constructed by using a radar image, and a position adjustment alignment interface is obtained through query, projection transformation and position adjustment; and identifying device parameters to analyze the interface offset, and further adjusting the interface offset to obtain a target alignment interface so as to realize the automatic alignment of the crane pipe. According to the invention, the precision of automatic alignment of the crane tube can be improved.

Description

Technical Field

[0001] The invention relates to the field of crane pipe alignment based on image analysis, and in particular to an automatic crane pipe alignment system based on image analysis. Background Art

[0002] As a key equipment in the process of oil loading and unloading, the accurate alignment of the crane pipe plays a vital role in the safe and efficient operation of oil loading and unloading. In oil loading and unloading operations, traditional crane pipe alignment methods mostly rely on manual operation. Operators align the crane pipe with the tank truck interface through visual observation and manual adjustment. This manual alignment method has many disadvantages. Not only is the alignment efficiency low, but also due to factors such as errors in human visual judgment and different operating proficiency, it is easy to have inaccurate alignment, which may lead to safety accidents such as oil leakage, posing a serious threat to the environment and personnel safety.

[0003] With the continuous development and progress of computer vision technology and image processing technology, image analysis methods have been widely used in various fields. At present, in the field of crane pipe alignment, attempts have been made to introduce related technologies based on image analysis to realize automatic alignment of crane pipes. By setting up appropriate image acquisition equipment in the loading and unloading area, real-time images of the interface between the crane pipe and the tank car are collected, and then the collected images are processed by the image analysis algorithm to identify the relative position information of the interface between the crane pipe and the tank car, and then the relevant mechanical devices are driven to automatically adjust the position of the crane pipe to achieve the purpose of accurate alignment. However, at this stage, there are still some challenges for the automatic alignment method of crane pipes based on image analysis. On the one hand, in the complex loading and unloading site environment, factors such as light changes and obstructions will interfere with the quality of image acquisition, affect the accuracy of subsequent image analysis, and cause deviations in the identified position information. On the other hand, the existing image analysis algorithm may not be accurate enough when processing the feature extraction and matching of specific targets such as the interface between the crane pipe and the tank car. Under some special working conditions, it is impossible to quickly and accurately judge the correct alignment situation, so that the accuracy of the automatic alignment of the crane pipe needs to be further improved. Summary of the invention

[0004] The present invention provides an automatic positioning system for crane pipes based on image analysis, the main purpose of which is to improve the accuracy of automatic positioning of crane pipes.

[0005] To achieve the above-mentioned purpose, the present invention provides an automatic crane pipe alignment system based on image analysis, comprising: an image acquisition module, a preliminary alignment module, an interface calibration module, an alignment adjustment module and a target alignment module;

[0006] The image acquisition module is used to acquire multiple types of images of the crane pipe to be aligned by using pre-configured multiple types of image acquisition equipment, wherein the multiple types of images include high-definition images, infrared images and radar images;

[0007] A preliminary alignment module is used to identify the spectral region of the high-definition image, perform a de-artifacting operation on the high-definition image based on the spectral region to obtain a de-artifacted image, identify the image semantics of the de-artifacted image, perform image segmentation on the de-artifacted image based on the image semantics to obtain a segmented image, and use the segmented image to identify the alignment interface of the crane pipe to be aligned to obtain a preliminary alignment interface;

[0008] An interface calibration module, used for identifying infrared pixels of the infrared image, calculating a domain drift vector mean of the infrared pixels, calculating a pixel inner product of the infrared pixels based on the domain drift vector mean, analyzing a pixel offset level of the infrared image based on the pixel inner product, identifying edge information of the infrared image based on the pixel offset level, and performing interface calibration on the preliminary alignment interface based on the edge information to obtain a calibrated interface;

[0009] An alignment adjustment module is used to construct a three-dimensional coordinate map of the crane pipe to be aligned using the radar image, query the interface position information of the calibration interface in the three-dimensional coordinate map, perform a projection transformation on the calibration interface based on the interface position information to obtain projection coordinates, and perform position adjustment on the calibration interface based on the projection coordinates to obtain a position-adjusted alignment interface;

[0010] The target alignment module is used to identify the device parameters of the crane pipe to be aligned, analyze the interface offset of the crane pipe to be aligned based on the device parameters, perform interface alignment adjustment on the position adjustment alignment interface based on the interface offset, obtain the target alignment interface, and automatically align the crane pipe to be aligned based on the target alignment interface.

[0011] Compared with the prior art, the embodiment of the present invention uses pre-configured multi-type image acquisition equipment to collect crane pipe images to be aligned, including high-definition images, infrared images and radar images. High-definition images can clearly show the appearance, position and other details of the crane pipe, which is conducive to the accurate identification of the alignment interface form; infrared images are based on thermal radiation imaging, highlighting the outline of the crane pipe under complex working conditions and capturing targets that are difficult to detect with conventional vision; radar images can accurately measure the spatial coordinates of the crane pipe and surrounding objects, etc., to provide key position data for alignment operations, and these images can be collected by corresponding high-definition cameras, infrared cameras and radar devices; further identify the spectral region of the high-definition image, determine its spectral composition and distribution, and then construct a filtering window, improve the image quality through filtering, evaluating visual feature indicators and image reconstruction and other de-artifacting operations, and identify the semantics of the de-artifacted image to understand the actual content represented by the pixels, and then perform corner detection, grayscale processing, non-maximum suppression, region division and segmentation to obtain a segmented image, and finally identify the preliminary alignment interface based on the characteristics of the crane pipe and its alignment interface, providing an initial reference for subsequent calibration and other operations; after identifying the infrared pixel points of the infrared image, through wavelet decomposition, image The mean of the drift vector in the field is calculated by sampling the prime points, and then the inner product of the pixel points is calculated based on this, the pixel offset level is analyzed, the edge information is identified by using the Canny algorithm, and then the feature descriptor is constructed for feature matching, and the preliminary alignment interface is calibrated to obtain the calibration interface, and the possible deviation of the preliminary alignment interface is corrected by using the more accurate edge information of the infrared image; the three-dimensional coordinate map of the crane pipe to be aligned is constructed by using the radar image, and the spatial coordinates are calculated by identifying the target information, querying the echo and posture information, and then the position information of the calibration interface in the map is clarified, and the projection coordinates are obtained through projection transformation, and then the calibration interface is moved in the three-dimensional space according to the adjustment parameters corresponding to the projection coordinates to obtain the position adjustment alignment interface, so as to improve the alignment accuracy; the device parameters of the crane pipe to be aligned, such as the length of the telescopic arm, are identified, and its kinematic characteristics are analyzed, and the interface offset is calculated, and the position adjustment alignment interface is adjusted according to the offset to obtain the target alignment interface, and finally the crane pipe is automatically aligned based on the target alignment interface. Therefore, the present invention can improve the accuracy of the automatic alignment of the crane pipe. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A functional module diagram of a crane pipe automatic alignment system based on image analysis provided by one embodiment of the present invention;

[0013] Figure 2 A schematic flow chart of a crane pipe automatic alignment method based on image analysis provided in one embodiment of the present invention;

[0014] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0017] In fact, the server-side device deployed by the crane pipe automatic alignment system based on image analysis may be composed of one or more devices. The above-mentioned crane pipe automatic alignment system based on image analysis can be implemented as: business instance, virtual machine, hardware device. For example, the crane pipe automatic alignment system based on image analysis can be implemented as a business instance deployed on one or more devices in the cloud node. In simple terms, the crane pipe automatic alignment system based on image analysis can be understood as a software deployed on the cloud node, which is used to provide each user terminal with the service of crane pipe automatic alignment based on image analysis. Alternatively, the crane pipe automatic alignment system based on image analysis can also be implemented as a virtual machine deployed on one or more devices in the cloud node. The virtual machine is installed with application software for managing each user terminal. Alternatively, the crane pipe automatic alignment system based on image analysis can also be implemented as a server composed of many hardware devices of the same or different types, and one or more hardware devices are set to provide each user terminal with the service of crane pipe automatic alignment based on image analysis.

[0018] In terms of implementation, the crane pipe automatic alignment system based on image analysis and the user end are adapted to each other. That is, the crane pipe automatic alignment system based on image analysis is an application installed on the cloud service platform, and the user end is a client that establishes a communication connection with the application; or the crane pipe automatic alignment system based on image analysis is implemented as a website, and the user end is implemented as a web page; or the crane pipe automatic alignment system based on image analysis is implemented as a cloud service platform, and the user end is implemented as a small program in an instant messaging application.

[0019] Reference Figure 1 , which is a functional module diagram of a crane pipe automatic alignment system based on image analysis provided in one embodiment of the present invention.

[0020] The crane pipe automatic alignment system 100 based on image analysis of the present invention can be set in a cloud server. In terms of implementation, it can be used as one or more service devices, or it can be installed as an application on the cloud (such as a server, server cluster, etc. for crane pipe automatic alignment based on image analysis), or it can be developed as a website. According to the implemented functions, the crane pipe automatic alignment system 100 based on image analysis includes an image acquisition module 101, a preliminary alignment module 102, an interface calibration module 103, an alignment adjustment module 104, and a target alignment module 105.

[0021] In an embodiment of the present invention, in the tracking of the automatic alignment of crane pipes based on image analysis, each of the above modules can be implemented independently and called with other modules. The call here can be understood as a module that can connect to multiple modules of another type and provide corresponding services to the multiple modules connected to it. In the automatic alignment system of crane pipes based on image analysis provided by an embodiment of the present invention, the scope of application of the automatic alignment architecture of crane pipes based on image analysis can be adjusted by adding modules and directly calling them without modifying the program code, thereby realizing cluster-based horizontal expansion, so as to achieve the purpose of quickly and flexibly expanding the automatic alignment system of crane pipes based on image analysis. In practical applications, the above modules can be set in the same device or different devices, or in virtual devices, such as service instances in cloud servers.

[0022] In the following, in conjunction with specific embodiments, each component and specific workflow of the crane pipe automatic alignment system based on image analysis are described respectively.

[0023] The image acquisition module 101 is used to acquire multiple types of images of the crane pipe to be aligned using pre-configured multiple types of image acquisition equipment, wherein the multiple types of images include high-definition images, infrared images and radar images.

[0024] The embodiment of the present invention can reflect the situation of the crane pipe and its surrounding environment from their own unique perspectives by collecting multiple types of images of the crane pipe to be aligned using the preconfigured multi-type image acquisition device. For example, high-definition images can clearly present detailed information such as the appearance structure, color, texture and relative position relationship of the crane pipe with surrounding objects, so that subsequent analysis can accurately identify the specific morphological features of the crane pipe itself and its alignment interface; infrared images are based on thermal radiation imaging of objects. In complex working conditions with poor light (such as at night, dim environment), obstruction or thermal interference (such as heating equipment, steam, etc. around), the outline formed by the temperature difference between the crane pipe and the surrounding environment is highlighted, which helps to capture target information that is difficult to detect with conventional vision; radar images can accurately measure the three-dimensional spatial coordinates, distance and other spatial position relationships of the crane pipe and surrounding objects, and provide key position data for subsequent accurate alignment operations in actual physical space.

[0025] Optionally, the multiple types of images can be collected by a high-definition camera, an infrared camera and a radar device.

[0026] The preliminary alignment module 102 is used to identify the spectral region of the high-definition image, perform a de-artifacting operation on the high-definition image based on the spectral region to obtain a de-artifacted image, identify the image semantics of the de-artifacted image, perform image segmentation on the de-artifacted image based on the image semantics to obtain a segmented image, and use the segmented image to identify the alignment interface of the crane pipe to be aligned to obtain a preliminary alignment interface.

[0027] The embodiment of the present invention can determine the spectral components and regional distribution corresponding to different wavelength ranges in the high-definition image by identifying the spectral region of the high-definition image, thereby enabling the user to clearly understand which spectrally related image content needs to be processed.

[0028] Among them, the spectral region refers to different intervals divided according to the wavelength or frequency range of light, such as the visible light region, infrared region and ultraviolet light region, etc. The light in each interval has specific properties and characteristics, such as wavelength range, wave performance and other characteristics.

[0029] Optionally, the spectral region can be identified by converting the high-definition image using a spectrometer.

[0030] The embodiment of the present invention performs a de-artifacting operation on the high-definition image based on the spectral region to obtain a de-artifacted image, which can improve the quality of the high-definition image, restore the image to a more realistic and accurate visual effect, and make the elements such as the crane pipe, alignment interface and surrounding environment in the image more clearly and clearly presented.

[0031] As an embodiment of the present invention, the de-artifacting operation is performed on the high-definition image based on the spectral region to obtain a de-artifacted image, including: constructing a filtering window for the high-definition image based on the spectral region, filtering the high-definition image using a low-pass filter based on the filtering window to obtain a filtered image, evaluating visual feature indicators of the filtered image, and reconstructing the filtered image when the visual feature indicators meet preset indicator values ​​to obtain a de-artifacted image.

[0032] Optionally, the filter window can be constructed by determining relevant parameters of the filter window according to the characteristics of the spectral region identified in the high-definition image, such as setting an appropriate horizontal width and vertical length according to the spectral distribution. The visual feature index can be set according to brightness, contrast and structural similarity, and the preset index value refers to the specific value of brightness, contrast and structural similarity set according to the image analysis requirements (need to be set in combination with the actual application).

[0033] The visual feature index refers to an image artifact removal evaluation index set based on the brightness, contrast and structural similarity of the image.

[0034] Optionally, constructing a filter window for the high-definition image based on the spectral region may be performed, and reconstructing the filtered image to obtain a de-artifacted image may be performed.

[0035] The embodiment of the present invention can understand the semantic information such as the actual objects represented by each pixel area in the de-artifacted image, the scene category and the relationship between them by identifying the image semantics of the de-artifacted image, that is, figure out what the different parts in the image correspond to, such as which part is the crane pipe and which part is the tank truck.

[0036] Optionally, the image semantics can be identified using a deep learning model.

[0037] Furthermore, the embodiment of the present invention performs image segmentation on the de-artifacted image based on the image semantics, and the obtained segmented image can separate the positioning interface of the crane pipe from the complex background and other irrelevant elements, highlighting the key target to improve the accuracy and efficiency of target recognition.

[0038] As an embodiment of the present invention, the de-artifacted image is segmented based on the image semantics to obtain a segmented image, including: performing corner point detection on the de-artifacted image based on the image semantics to obtain detected corner points, converting the de-artifacted image into a grayscale image, performing non-maximum suppression on the grayscale image to obtain a suppressed image, dividing the suppressed image into areas to be segmented using the detected corner points, calculating the mean intersection and union ratio of the area to be segmented, and performing region segmentation on the area to be segmented based on the mean intersection and union ratio to obtain a segmented image.

[0039] Optionally, performing corner detection on the de-artifacted image based on the image semantics to obtain the detected corners can be implemented using a corner detection function. Performing non-maximum suppression on the grayscale image to obtain the suppressed image can be implemented using a non-maximum suppression algorithm.

[0040] Among them, the mean intersection union ratio refers to a measurement indicator commonly used in computer vision tasks such as image segmentation and target detection, which is used to evaluate the degree of overlap between the predicted area (such as the area where the target object is predicted by the algorithm) and the true labeled area (the accurate area where the actual object is located, usually determined as a standard by manual labeling, etc.).

[0041] Furthermore, the embodiment of the present invention utilizes the segmented image to identify the alignment interface of the crane pipe to be aligned, and obtains a preliminary alignment interface, so that the specific location of the alignment interface of the crane pipe can be found in the image, which is convenient for the user to mark its boundaries, center points and other key position information, and determine the preliminary alignment interface situation. At the same time, it also provides an initial and relatively accurate target reference for subsequent further calibration and adjustment of the crane pipe position.

[0042] Optionally, the segmented image is used to identify the alignment interface of the crane pipe to be aligned, and the preliminary alignment interface can be searched and compared in the segmented image based on the known features (such as shape, texture, etc.) of the crane pipe and its alignment interface using a feature matching or template matching algorithm. Then, the area that meets the alignment interface features is screened out, and its key position information such as the boundary and center is marked, so as to identify the alignment interface of the crane pipe to be aligned and determine it as the preliminary alignment interface.

[0043] The interface calibration module 103 is used to identify the infrared pixels of the infrared image, calculate the domain drift vector mean of the infrared pixels, calculate the pixel inner product of the infrared pixels based on the domain drift vector mean, analyze the pixel offset level of the infrared image based on the pixel inner product, identify the edge information of the infrared image based on the pixel offset level, and perform interface calibration on the preliminary alignment interface based on the edge information to obtain a calibrated interface.

[0044] The embodiment of the present invention can obtain the thermal radiation information of the objects in the image by identifying the infrared pixels of the infrared image, so that the user can mark them out from the entire image data structure for subsequent targeted analysis and processing. The infrared pixels refer to the basic elements constituting the infrared image.

[0045] Optionally, the infrared pixel points can be identified by spectral characteristics.

[0046] The embodiment of the present invention can discover key position information such as object boundaries that may exist in the image by calculating the domain drift vector mean of the infrared pixel points, and further discover some interference factors in the infrared image due to the edge of the object, temperature gradient changes, and the imaging process.

[0047] The domain drift vector mean refers to an indicator that reflects the change in the spatial position of the neighborhood pixel points relative to the central pixel point.

[0048] As an embodiment of the present invention, the calculating the mean value of the domain drift vector of the infrared pixel point includes: performing wavelet decomposition on the infrared image corresponding to the infrared pixel point to obtain a decomposed image, and calculating the mean value of the domain drift vector of the sampling pixel point using the following formula:

[0049]

[0050] Among them, a represents the mean of the domain drift vector, n represents the number of sampling pixels, and W max represents the maximum value of the spatial dimension of the decomposed image, K represents the pixel density gradient of the decomposed image, A represents the coordinates of a target pixel in the sampling pixels, A1 represents the coordinates of the domain pixel of A, and (A1-A) represents the floating vector of A relative to A1.

[0051] Optionally, performing wavelet decomposition on the infrared image corresponding to the infrared pixel point to obtain a decomposed image may be achieved by utilizing a wavelet decomposition function.

[0052] The embodiment of the present invention can measure the consistency of pixel shift by calculating the pixel inner product of the infrared pixel based on the domain drift vector mean, so as to analyze the structural change of the image.

[0053] The pixel inner product refers to a scalar that measures pixel characteristics from multiple angles.

[0054] As an embodiment of the present invention, the step of calculating the inner product of the infrared pixel points based on the domain drift vector mean value includes: calculating the inner product of the infrared pixel points based on the domain drift vector mean value using the following formula:

[0055]

[0056] Among them, β represents the inner product of the pixel, α represents the mean of the domain drift vector, δ represents the pixel probability density of the infrared pixel, γ represents the color contribution of the infrared pixel, r represents the pixel illumination of the infrared pixel, and u v It represents the corresponding position of the infrared pixel v in the decomposed image space domain, and T represents a target pixel of the infrared pixel.

[0057] The corresponding position of the point in the decomposed image space domain, x is a target pixel in the infrared pixel points.

[0058] The embodiment of the present invention can evaluate and analyze the pixel offset level of the infrared image based on the inner product of the pixel points from the overall perspective, so as to understand whether there is uneven pixel movement in the image due to object movement, thermal radiation changes, imaging interference, etc.

[0059] Optionally, the pixel offset level of the infrared image based on the inner product of the pixel points can be analyzed by counting the inner product values ​​of each pixel point in the infrared image, observing their distribution in different areas of the image, and then, based on the size of the inner product value and the law of distribution change, for example, areas with large inner products often have large pixel offsets, judging the relative stability or offset degree of pixels in each area, thereby analyzing the pixel offset level of the entire image.

[0060] The embodiment of the present invention can accurately identify the edge information of the infrared image such as the edge lines and boundary contours of the crane pipe, alignment interface and surrounding objects in the infrared image by identifying the edge information of the infrared image based on the pixel offset level, and clearly separate the object from the background and adjacent parts.

[0061] Optionally, the identifying edge information of the infrared image based on the pixel offset level may be implemented using a Canny algorithm.

[0062] The embodiment of the present invention performs interface calibration on the preliminary alignment interface based on the edge information, and the calibration interface can use the accurate boundary and contour information represented by the edge to correct and adjust the position, shape, etc. of the preliminary alignment interface. For example, if the preliminary alignment interface has deviations in high-definition image analysis due to certain artifacts or inaccurate feature recognition, then the more accurate edge information in the infrared image can be used to adjust it to the correct position that matches the edge of the actual object, further improving the accuracy of the alignment interface position.

[0063] As an embodiment of the present invention, the interface calibration is performed on the preliminary alignment interface based on the edge information to obtain a calibrated interface, including: identifying the infrared image edge point coordinates, the infrared image edge direction and the infrared image edge pixel gradient of the infrared image based on the edge information, constructing a first feature descriptor of the preliminary alignment interface using the infrared image edge point coordinates, the infrared image edge direction and the infrared image edge pixel gradient, querying the high-definition edge image point coordinates, the high-definition image edge direction and the high-definition image edge pixel gradient of the high-definition image corresponding to the preliminary alignment interface, constructing a second feature descriptor of the preliminary alignment interface using the high-definition image edge point coordinates, the high-definition image edge direction and the high-definition image edge pixel gradient, performing feature matching on the first feature descriptor and the second feature descriptor, and performing position parameter calibration on the preliminary alignment interface based on the feature matching to obtain a calibrated interface.

[0064] Optionally, the infrared image edge point coordinates, infrared image edge direction and infrared image edge pixel gradient of the infrared image based on the edge information can be identified by an image analysis algorithm, searching for edge pixels in the infrared image, determining their coordinate positions, then analyzing the edge direction to obtain the edge direction, and calculating the pixel grayscale change to obtain the edge pixel gradient. The first feature descriptor and the second feature descriptor can be constructed using the HOG algorithm.

[0065] The alignment adjustment module 104 is used to use the radar image to construct a three-dimensional coordinate map of the crane pipe to be aligned, query the interface position information of the calibration interface in the three-dimensional coordinate map, perform a projection transformation on the calibration interface based on the interface position information to obtain projection coordinates, and adjust the position of the calibration interface based on the projection coordinates to obtain a position-adjusted alignment interface.

[0066] The embodiment of the present invention utilizes the radar image to construct a three-dimensional coordinate diagram of the crane pipe to be aligned, so as to intuitively and accurately identify the shape, posture and relative position relationship of the crane pipe with the surrounding environment in the actual physical space.

[0067] As an embodiment of the present invention, the use of the radar image to construct a three-dimensional coordinate map of the crane pipe to be aligned includes: identifying target information of the radar image, querying echo information and posture information of the radar corresponding to the radar image, identifying distance characteristics, shape characteristics and size characteristics of the target information based on the echo information, identifying angle characteristics of the target information based on the posture information, calculating the spatial coordinates of the target information based on the distance characteristics, the shape characteristics, the size characteristics and the angle characteristics, drawing a three-dimensional coordinate map of the target information, and constructing a three-dimensional coordinate map of the crane pipe to be aligned based on the spatial coordinates and the three-dimensional coordinate map.

[0068] Among them, the echo information refers to the relevant data carried by the wave reflected back after the electromagnetic wave emitted by the radar encounters the target object, and the posture information refers to the position and posture of the radar device itself in space, such as position coordinates, azimuth and pitch angle.

[0069] Optionally, the target information can be identified using a target recognition algorithm, the echo information and the posture information can be obtained using the storage in the radar device, and the distance characteristics, shape characteristics and size characteristics of the target information can be analyzed based on the intensity, delay and other conditions of the echo to obtain the distance characteristics corresponding to the target information, and then the shape characteristics and size characteristics can be determined by the contour reflected by the echo. The angle characteristics can be combined with the radar posture information and the presentation of the target in the radar image to deduce the angle characteristics of the target information in the horizontal, vertical and other directions. The spatial coordinates can be calculated using the target information combined with the spatial geometry algorithm. The three-dimensional coordinate diagram can be constructed using the openGL tool.

[0070] The embodiment of the present invention can clarify the position information of the calibration interface in the three-dimensional coordinate map by using the three-dimensional coordinate map to identify the interface position information of the calibration interface, and can provide a reliable reference basis for subsequent position adjustment.

[0071] Optionally, the interface location information may be obtained by querying coordinate information of the three-dimensional coordinate graph.

[0072] Furthermore, the embodiment of the present invention can intuitively observe the relative position deviation between the calibration interface and the target position by performing a projection transformation on the calibration interface based on the interface position information, so as to further accurately derive the position adjustment parameters required.

[0073] As an embodiment of the present invention, the projection transformation of the calibration interface to obtain the projection coordinates includes: determining the original horizontal axis coordinates and the original vertical axis coordinates of the calibration interface based on the interface position information, scaling and rotating the calibration interface based on the original horizontal axis coordinates and the original vertical axis coordinates, calculating the original horizontal axis coordinates and the original vertical axis coordinates using the following formula, and obtaining the projection coordinates by changing the coordinates after the scaling and rotation operations:

[0074]

[0075] Among them, x′ represents the horizontal coordinate of the projection coordinate, y′ represents the vertical coordinate of the projection coordinate, x represents the original horizontal axis coordinate, y represents the original vertical axis coordinate, θ represents the rotation angle, s x Indicates the scaling ratio of the horizontal axis, s y Indicates the scaling ratio of the vertical axis, Q x Indicates the translation amount in the original horizontal axis direction, Q y Indicates the amount of translation in the original horizontal axis direction.

[0076] Furthermore, the embodiment of the present invention can further improve the accuracy of the alignment of the crane pipe interface by adjusting the position of the calibration interface based on the projection coordinates to obtain a position-adjusted alignment interface.

[0077] Optionally, the calibration interface is position-adjusted based on the projection coordinates, and the position-adjusted alignment interface can be moved in three-dimensional space according to the distance, angle and other changes specified by the adjustment parameters corresponding to the projection coordinates, so that it gradually approaches and finally accurately aligns with the target docking position. The target alignment module 105 is used to identify the device parameters of the crane pipe to be aligned, analyze the interface offset of the crane pipe to be aligned based on the device parameters, and perform interface alignment adjustment on the position-adjusted alignment interface based on the interface offset to obtain a target alignment interface, and automatically align the crane pipe to be aligned based on the target alignment interface.

[0078] The embodiment of the present invention can understand the range of motion, motion accuracy and coordination between different actions of the crane pipe in three-dimensional space by identifying the device parameters of the crane pipe to be aligned.

[0079] The device parameters include the telescopic arm length range of the crane tube, the current extension length, the rotation angle range of each joint, the current angle value, and the translational travel range.

[0080] Optionally, the alignment device parameters can be obtained by querying the device manual of the crane pipe to be aligned.

[0081] Furthermore, the embodiment of the present invention can clarify the gap between the current alignment state and the ideal alignment state by analyzing the interface offset of the crane pipe to be aligned based on the device parameters, and convert the abstract alignment situation into a specific and quantifiable deviation value, so that subsequent adjustment operations have clear goals and directions.

[0082] As an embodiment of the present invention, the analysis of the interface offset of the crane pipe to be aligned when it is aligned based on the device parameters includes: querying the current position of the crane pipe to be aligned and the interface to be aligned corresponding to the crane pipe to be aligned to obtain a first position parameter and a second position number, unifying the coordinate system of the first position and the second position to obtain a first coordinate and a second coordinate, analyzing the kinematic characteristics of the crane pipe to be aligned based on the device parameters, analyzing the coordinate position offset of the first coordinate of the crane pipe to be aligned when it is aligned based on the kinematic characteristics, and calculating the interface offset of the crane pipe to be aligned when it is aligned based on the coordinate position offset and the second coordinate.

[0083] The kinematic characteristics refer to a series of properties and characteristics involved in describing the movement of an object in space, such as the degree of freedom of movement, range of movement, and accuracy of movement, etc. The interface to be aligned refers to the interface of the crane to be aligned and the interface of the oil truck to be used for oil transportation.

[0084] Optionally, the kinematic characteristics of the crane pipe to be aligned based on the device parameters analysis can be achieved by carefully studying the device parameters of the crane pipe, such as the telescopic range of the telescopic arm, the rotation angle range of each joint, the translation stroke range and the motion accuracy of each driving component, etc., to understand what kind of movement the crane pipe can perform in three-dimensional space and the limits and accuracy of the movement, clarify its adjustable dimensions and capacity range, so as to master the kinematic characteristics of the crane pipe.

[0085] Furthermore, the embodiment of the present invention performs interface alignment adjustment on the position adjustment alignment interface based on the interface offset to obtain a target alignment interface so that the position and posture of the preliminary alignment interface can gradually approach and eventually reach an ideal alignment state to ensure the accuracy of the final alignment.

[0086] Optionally, the principle of performing interface alignment adjustment on the position adjustment alignment interface based on the interface offset to obtain the target alignment interface is the same as the step of performing position adjustment on the calibration interface based on the projection coordinates to obtain the position adjustment alignment interface, so it will not be elaborated herein.

[0087] The embodiment of the present invention realizes the complete operation of automatic alignment of the crane pipe by automatically aligning the crane pipe to be aligned based on the target alignment interface, thereby ensuring that the crane pipe can stably and accurately cooperate with the docking target during the entire loading and unloading operation.

[0088] like Figure 2 FIG. 1 is a flow chart of an automatic positioning method for a crane pipe based on image analysis provided by an embodiment of the present invention. In this embodiment, the automatic positioning method for a crane pipe based on image analysis includes:

[0089] A preconfigured multi-type image acquisition device is used to acquire multi-type images of the crane pipe to be aligned, wherein the multi-type images include high-definition images, infrared images and radar images.

[0090] Identify the spectral region of the high-definition image, perform a de-artifacting operation on the high-definition image based on the spectral region to obtain a de-artifacted image, identify the image semantics of the de-artifacted image, perform image segmentation on the de-artifacted image based on the image semantics to obtain a segmented image, and use the segmented image to identify the alignment interface of the crane pipe to be aligned to obtain a preliminary alignment interface.

[0091] Identify the infrared pixels of the infrared image, calculate the domain drift vector mean of the infrared pixels, calculate the pixel inner product of the infrared pixels based on the domain drift vector mean, analyze the pixel offset level of the infrared image based on the pixel offset level, identify the edge information of the infrared image based on the pixel offset level, and perform interface calibration on the preliminary alignment interface based on the edge information to obtain a calibrated interface.

[0092] Use the radar image to construct a three-dimensional coordinate map of the crane pipe to be aligned, query the interface position information of the calibration interface in the three-dimensional coordinate map, perform projection transformation on the calibration interface based on the interface position information to obtain projection coordinates, and adjust the position of the calibration interface based on the projection coordinates to obtain a position-adjusted alignment interface.

[0093] Identify the device parameters of the crane pipe to be aligned, analyze the interface offset of the crane pipe to be aligned based on the device parameters, perform interface alignment adjustment on the position adjustment alignment interface based on the interface offset to obtain a target alignment interface, and automatically align the crane pipe to be aligned based on the target alignment interface.

[0094] In the several embodiments provided by the present invention, it should be understood that the provided system and method can be implemented in other ways. For example, the system embodiment described above is only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0095] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. An automatic positioning system for crane pipes based on image analysis, characterized in that: The system for automatically aligning crane pipes based on image analysis includes: an image acquisition module, a preliminary alignment module, an interface calibration module, an alignment adjustment module and a target alignment module; The image acquisition module is used to acquire multiple types of images of the crane pipe to be aligned by using pre-configured multiple types of image acquisition equipment, wherein the multiple types of images include high-definition images, infrared images and radar images; The preliminary alignment module is used to identify the spectral region of the high-definition image, perform a de-artifacting operation on the high-definition image based on the spectral region to obtain a de-artifacted image, identify the image semantics of the de-artifacted image, perform image segmentation on the de-artifacted image based on the image semantics to obtain a segmented image, and use the segmented image to identify the alignment interface of the crane pipe to be aligned to obtain a preliminary alignment interface; The interface calibration module is used to identify infrared pixels of the infrared image, calculate the domain drift vector mean of the infrared pixels, calculate the pixel inner product of the infrared pixels based on the domain drift vector mean, analyze the pixel offset level of the infrared image based on the pixel inner product, identify the edge information of the infrared image based on the pixel offset level, and perform interface calibration on the preliminary alignment interface based on the edge information to obtain a calibrated interface; The alignment adjustment module is used to construct a three-dimensional coordinate map of the crane pipe to be aligned using the radar image, query the interface position information of the calibration interface in the three-dimensional coordinate map, perform projection transformation on the calibration interface based on the interface position information to obtain projection coordinates, and perform position adjustment on the calibration interface based on the projection coordinates to obtain a position-adjusted alignment interface; The target alignment module is used to identify the device parameters of the crane pipe to be aligned, analyze the interface offset of the crane pipe to be aligned based on the device parameters, perform interface alignment adjustment on the position adjustment alignment interface based on the interface offset, obtain the target alignment interface, and automatically align the crane pipe to be aligned based on the target alignment interface.

2. The crane pipe automatic alignment system based on image analysis as claimed in claim 1, characterized in that: The performing a de-artifacting operation on the high-definition image based on the spectral region to obtain a de-artifacted image includes: Based on the spectral region, construct a filtering window for the high-definition image; Based on the filtering window, using a low-pass filter to filter the high-definition image to obtain a filtered image; Evaluating visual feature indicators of the filtered image; When the visual feature index meets a preset index value, the filtered image is reconstructed to obtain a de-artifacted image.

3. The crane pipe automatic alignment system based on image analysis as claimed in claim 1, characterized in that: The performing image segmentation on the de-artifacted image based on the image semantics to obtain a segmented image includes: Based on the image semantics, performing corner point detection on the artifact-free image to obtain detected corner points; Converting the de-artifacted image into a grayscale image; Performing non-maximum suppression on the grayscale image to obtain a suppressed image; Dividing the to-be-segmented area of ​​the suppressed image by using the detected corner points; Calculate the mean intersection and union ratio of the area to be segmented; Based on the mean intersection and union ratio, the region to be segmented is segmented to obtain a segmented image.

4. The crane pipe automatic alignment system based on image analysis as claimed in claim 1, characterized in that: The calculating the mean of the domain drift vector of the infrared pixel point comprises: Performing wavelet decomposition on the infrared image corresponding to the infrared pixel point to obtain a decomposed image; Performing pixel sampling on the decomposed image to obtain sampled pixel points; Calculate the mean value of the area drift vector of the sampling pixel point.

5. The crane pipe automatic alignment system based on image analysis as claimed in claim 1, characterized in that: The step of calculating the inner product of the infrared pixel points based on the domain drift vector mean value includes: Based on the domain drift vector mean, the pixel inner product of the infrared pixel is calculated.

6. The crane pipe automatic alignment system based on image analysis as claimed in claim 1, characterized in that: The step of performing interface calibration on the preliminary alignment interface based on the edge information to obtain a calibrated interface includes: Identify the infrared image edge point coordinates, infrared image edge direction and infrared image edge pixel gradient of the infrared image based on the edge information; Constructing a first feature descriptor of the preliminary alignment interface by using the infrared image edge point coordinates, the infrared image edge direction and the infrared image edge pixel gradient; Querying the high-definition edge image point coordinates, high-definition image edge direction and high-definition image edge pixel gradient of the high-definition image corresponding to the preliminary alignment interface; Using the edge point coordinates of the high-definition image, the edge direction of the high-definition image, and the edge pixel gradient of the high-definition image, a second feature descriptor for constructing the preliminary alignment interface is constructed; Performing feature matching on the first feature descriptor and the second feature descriptor; Based on the feature matching, position parameters of the preliminary alignment interface are calibrated to obtain a calibrated interface.

7. The crane pipe automatic alignment system based on image analysis as claimed in claim 1, characterized in that: The method of using the radar image to construct a three-dimensional coordinate diagram of the crane pipe to be aligned includes: Identifying target information of the radar image; Querying the echo information and position information of the radar corresponding to the radar image; Identify the distance feature, shape feature and size feature of the target information based on the echo information; Identify the angle feature of the target information based on the posture information; Calculating the spatial coordinates of the target information based on the distance feature, the shape feature, the size feature and the angle feature; Draw a three-dimensional coordinate diagram of the target information; Based on the spatial coordinates and the three-dimensional coordinate diagram, a three-dimensional coordinate diagram of the crane pipe to be aligned is constructed.

8. The crane pipe automatic alignment system based on image analysis as claimed in claim 1, characterized in that: The performing projection transformation on the calibration interface to obtain projection coordinates includes: Based on the interface position information, determining the original horizontal axis coordinate and the original vertical axis coordinate of the calibration interface; Based on the original horizontal axis coordinate and the original vertical axis coordinate, scaling and rotating the calibration interface; The original horizontal axis coordinate and the original vertical axis coordinate are calculated using the following formula, and the changed coordinates after the scaling and rotation operations are performed to obtain the projection coordinates.

9. The crane pipe automatic alignment system based on image analysis as claimed in claim 1, characterized in that: The analyzing the interface offset of the crane pipe to be aligned based on the device parameters includes: Query the current position of the crane pipe to be aligned and the interface to be aligned corresponding to the crane pipe to be aligned to obtain a first position parameter and a second position number; Unifying the coordinate systems of the first position and the second position to obtain a first coordinate and a second coordinate; Analyzing the kinematic characteristics of the crane pipe to be aligned based on the device parameters; Based on the kinematic characteristics, analyzing the coordinate position offset of the first coordinate when the crane pipe to be aligned is aligned; Based on the coordinate position offset and the second coordinate, the interface offset of the crane pipe to be aligned is calculated.

10. A crane pipe automatic alignment method based on image analysis, characterized in that: The method comprises: Using a pre-configured multi-type image acquisition device to acquire multi-type images of the crane pipe to be aligned, wherein the multi-type images include high-definition images, infrared images, and radar images; Identify the spectral region of the high-definition image, perform a de-artifacting operation on the high-definition image based on the spectral region to obtain a de-artifacting image, identify the image semantics of the de-artifacting image, perform image segmentation on the de-artifacting image based on the image semantics to obtain a segmented image, and use the segmented image to identify the alignment interface of the crane pipe to be aligned to obtain a preliminary alignment interface; Identify infrared pixels of the infrared image, calculate a domain drift vector mean of the infrared pixel, calculate a pixel inner product of the infrared pixel based on the domain drift vector mean, analyze a pixel offset level of the infrared image based on the pixel inner product, identify edge information of the infrared image based on the pixel offset level, and perform interface calibration on the preliminary alignment interface based on the edge information to obtain a calibrated interface; Using the radar image, construct a three-dimensional coordinate map of the crane pipe to be aligned, query the interface position information of the calibration interface in the three-dimensional coordinate map, perform projection transformation on the calibration interface based on the interface position information to obtain projection coordinates, and adjust the position of the calibration interface based on the projection coordinates to obtain a position-adjusted alignment interface; Identify the device parameters of the crane pipe to be aligned, analyze the interface offset of the crane pipe to be aligned based on the device parameters, perform interface alignment adjustment on the position adjustment alignment interface based on the interface offset to obtain a target alignment interface, and automatically align the crane pipe to be aligned based on the target alignment interface.

Images