Intelligent collaborative anti-collision method and system for tower groups

Abstract

The embodiment of the present invention provides a group tower intelligent collaborative anti-collision method and system, which belongs to the technical field of tower crane operation control. The method includes: determining the first edge information of the first tower crane according to the first image data and the first sensor data; determining the second edge information of the second tower crane according to the second image data and the second sensor data; obtaining the first position information of the first tower crane by using the Beidou positioning system and obtaining the second position information of the second tower crane by using the Beidou positioning system; determining the first boundary information of the first tower crane according to the first edge information and the first position information; determining the second boundary information of the second tower crane according to the second edge information and the second position information; determining the target operation state of the first tower crane and the second tower crane according to the group tower operation rules combined with the first boundary information and the second boundary information; performing anti-collision adjustment on the first tower crane and the second tower crane according to the target operation state, and obtaining the target processing results of the first tower crane and the second tower crane.

Description

Technical Field

[0001] The present invention relates to the technical field of tower crane operation control, and in particular to a group-tower intelligent collaborative anti-collision method and system. Background Art

[0002] In modern large-scale construction projects, in order to improve construction efficiency, multiple tower cranes are usually deployed in actual scenarios for simultaneous operation. The parallel operation of multiple tower cranes not only speeds up the progress of the project, but also optimizes resource allocation to a certain extent. However, the collaborative operation of multiple tower cranes in the same construction area also brings potential safety hazards, especially the risk of collision. Therefore, in complex construction environments, the use of intelligent collaboration and anti-collision control technology has become an important means to ensure the safety and efficiency of group tower operations. Although intelligent collaboration and anti-collision control technology have shown great potential in ensuring construction safety and efficiency, its practical application still faces many technical challenges and development bottlenecks. When the existing technology detects a possible collision, it often needs to rely on the operator's experience to avoid the tower crane. This method of relying on manual judgment not only increases the burden on the operator, but also may cause delays in avoidance decisions, thereby increasing the risk of collision. In addition, when avoiding tower cranes, operators need to communicate with each other to determine the avoidance object, which not only increases the complexity of the operation, but also may lead to operational errors. Especially in a tense construction environment, operators may make wrong avoidance decisions due to poor communication or information asymmetry, thereby causing safety accidents, and the existing technology also further reduces the efficiency of tower crane operation. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a method and system for intelligent collaborative collision avoidance among tower groups, aiming to solve the problem in related technologies that when a possible collision is detected, it is often necessary to rely on the operator's experience to avoid the tower crane. This reliance on manual judgment not only increases the burden on the operator, but may also lead to delays in avoidance decisions, thereby increasing the risk of collision.

[0004] In a first aspect, an embodiment of the present invention provides a group-tower intelligent collaborative collision avoidance method, comprising:

[0005] Using a first camera to collect first image data corresponding to a first tower crane and using a first sensor to collect first sensor data corresponding to the first tower crane;

[0006] Using a second camera to collect second image data corresponding to a second tower crane and using a second sensor to collect second sensor data corresponding to the second tower crane;

[0007] Determine first edge information corresponding to the first tower crane according to the first image data and the first sensor data;

[0008] Determine second edge information corresponding to the second tower crane according to the second image data and the second sensor data;

[0009] Using the Beidou positioning system to obtain first position information corresponding to the first tower crane and using the Beidou positioning system to obtain second position information corresponding to the second tower crane;

[0010] Determine first boundary information corresponding to the first tower crane according to the first edge information and the first position information;

[0011] Determine second boundary information corresponding to the second tower crane according to the second edge information and the second position information;

[0012] Determine a tower group operation rule, and determine a target operation state corresponding to the first tower crane and the second tower crane according to the tower group operation rule combined with the first boundary information and the second boundary information;

[0013] Anti-collision adjustments are performed on the first tower crane and the second tower crane according to the target operating state to obtain target processing results of the first tower crane and the second tower crane.

[0014] In a second aspect, an embodiment of the present invention provides a group-tower intelligent collaborative collision avoidance system, comprising:

[0015] A first acquisition module, used to acquire first image data corresponding to the first tower crane using a first camera and to acquire first sensor data corresponding to the first tower crane using a first sensor;

[0016] A second acquisition module, used to acquire second image data corresponding to the second tower crane using a second camera and to acquire second sensor data corresponding to the second tower crane using a second sensor;

[0017] A first processing module, configured to determine first edge information corresponding to the first tower crane according to the first image data and the first sensor data;

[0018] A second processing module, used for determining second edge information corresponding to the second tower crane according to the second image data and the second sensor data;

[0019] A position determination module, used to obtain first position information corresponding to the first tower crane using a Beidou positioning system and to obtain second position information corresponding to the second tower crane using the Beidou positioning system;

[0020] A third processing module, configured to determine first boundary information corresponding to the first tower crane according to the first edge information and the first position information;

[0021] a fourth processing module, configured to determine second boundary information corresponding to the second tower crane according to the second edge information and the second position information;

[0022] A state determination module, used to determine a tower group operation rule, and determine a target operation state corresponding to the first tower crane and the second tower crane according to the tower group operation rule combined with the first boundary information and the second boundary information;

[0023] The data adjustment module is used to perform anti-collision adjustment on the first tower crane and the second tower crane according to the target operating state to obtain target processing results of the first tower crane and the second tower crane.

[0024] The embodiment of the present invention provides a method and system for intelligent collaborative anti-collision of a group of towers, the method comprising: using a first camera to collect first image data corresponding to a first tower crane and using a first sensor to collect first sensor data corresponding to the first tower crane; using a second camera to collect second image data corresponding to a second tower crane and using a second sensor to collect second sensor data corresponding to the second tower crane; determining first edge information corresponding to the first tower crane according to the first image data and the first sensor data; determining second edge information corresponding to the second tower crane according to the second image data and the second sensor data; using a Beidou positioning system to obtain first position information corresponding to the first tower crane and using a Beidou positioning system to obtain second position information corresponding to the second tower crane; determining first boundary information corresponding to the first tower crane according to the first edge information and the first position information; determining second boundary information corresponding to the second tower crane according to the second edge information and the second position information; determining a group tower operation rule, and determining a target operation state corresponding to the first tower crane and the second tower crane according to the group tower operation rule in combination with the first boundary information and the second boundary information; performing anti-collision adjustment on the first tower crane and the second tower crane according to the target operation state to obtain target processing results of the first tower crane and the second tower crane. The method collects the first image data and the first sensor data of the first tower crane respectively through the first camera and the first sensor, and collects the second image data and the second sensor data of the second tower crane respectively through the second camera and the second sensor, so as to obtain comprehensive and highly accurate tower crane status information. The combination of such multi-source data can provide more comprehensive and accurate tower crane operation status data, and provide a reliable basis for subsequent anti-collision decisions. The Beidou positioning system is then used to obtain the accurate first position information and the second position information of the first tower crane and the second tower crane, so as to ensure the accuracy and real-time performance of the tower crane position data. Such high-precision position information is the key to achieving accurate anti-collision adjustment, and helps to avoid the collision risk caused by position error. According to the target operation state, the tower crane is adjusted in real time for anti-collision, so as to correct the unsafe operation of the tower crane in time and avoid collision accidents caused by operation delay. Such real-time adjustment capability can significantly improve the efficiency of anti-collision control and reduce potential safety risks. By making accurate anti-collision adjustment to the tower crane, the target processing result of the tower crane is finally obtained, so as to ensure that each tower crane always maintains a safe operation state during the collaborative operation. Such optimized processing result helps to improve the overall construction efficiency and reduce the time and economic losses caused by collision accidents. This will ensure the safe and efficient operation of tower cranes in complex construction environments, and provide strong support for the safe and smooth implementation of large-scale construction projects. It also solves the problem in related technologies that when a possible collision is detected, the tower crane often needs to rely on the operator's experience to avoid the tower crane. This reliance on manual judgment not only increases the burden on the operator, but may also lead to delays in avoidance decisions, thereby increasing the risk of collision. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A schematic diagram of a flow chart of a group-tower intelligent collaborative anti-collision method provided by an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the module structure of a group-tower intelligent collaborative collision avoidance system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0030] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0031] The embodiment of the present invention provides a method and system for intelligent collaborative collision avoidance of a group of towers. The method can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0032] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] Please refer to Figure 1 , Figure 1 A schematic flow chart of a method for intelligent collaborative anti-collision among tower groups provided in an embodiment of the present invention.

[0034] like Figure 1 As shown, the group-tower intelligent collaborative collision avoidance method includes steps S101 to S109.

[0035] Step S101: using a first camera to collect first image data corresponding to a first tower crane and using a first sensor to collect first sensor data corresponding to the first tower crane.

[0036] For example, a high-resolution industrial camera or surveillance camera is selected to ensure the clarity and accuracy of the first image data. The first camera is installed at a key position of the first tower crane, such as the tower crane arm end or near the cab, to ensure that the operating status and surrounding environment of the first tower crane can be fully captured. The first camera is fixed to the first tower crane using a stable bracket and preliminary adjustments are made to ensure that the field of view of the first camera covers the entire working area and the image is clear and unobstructed.

[0037] Exemplarily, the first sensor may be a lidar sensor, so that the first sensor is installed at a key position of the first tower crane, such as a tower crane arm, a cab or a lifting device, to ensure that relevant physical quantities can be accurately measured.

[0038] Exemplarily, the first camera is started to collect the surrounding image data of the first tower crane in real time. It is ensured that the first camera can continuously and stably capture the first image data of the first tower crane. At the same time, the first sensor is started to collect the first sensor data of the first tower crane in real time. It is ensured that the sensor can accurately and stably measure the relevant physical quantity.

[0039] Step S102: using a second camera to collect second image data corresponding to a second tower crane and using a second sensor to collect second sensor data corresponding to the second tower crane.

[0040] For example, a high-resolution industrial camera or surveillance camera is selected to ensure the clarity and accuracy of the second image data. The second camera is installed at a key position of the second tower crane, such as the tower crane arm end or near the cab, to ensure that the operating status and surrounding environment of the second tower crane can be fully captured. The second camera is fixed to the second tower crane using a stable bracket and preliminary adjustments are made to ensure that the second camera's field of view covers the entire working area and the image is clear and unobstructed.

[0041] Exemplarily, the second sensor may be a lidar sensor, so that the second sensor is installed at a key position of the second tower crane, such as a tower crane arm, a cab or a lifting device, to ensure that relevant physical quantities can be accurately measured.

[0042] Exemplarily, the second camera is started to collect the surrounding image data of the second tower crane in real time. It is ensured that the second camera can continuously and stably capture the second image data of the second tower crane. At the same time, the second sensor is started to collect the second sensor data of the second tower crane in real time. It is ensured that the sensor can accurately and stably measure the relevant physical quantity.

[0043] Step S103: Determine first edge information corresponding to the first tower crane according to the first image data and the first sensor data.

[0044] Exemplarily, a suitable edge detection algorithm is selected, such as Canny edge detection, Sobel operator, Laplacian operator, etc. Parameters of the edge detection algorithm are set according to the first image data, such as high and low thresholds, convolution kernel size, etc. Ensure that the algorithm can detect the first image edge information corresponding to the first target object in the first image data. And refine the detected edge, remove redundant information, and enhance the clarity and continuity of the edge. Morphological processing (such as dilation, corrosion) and other techniques can be used to optimize the edge.

[0045] Exemplarily, feature information related to the edge information of the first image is extracted from the first sensor data, and then the extracted sensor data features are matched with the edge information of the first image to ensure the consistency and relevance of the information between the two, so as to fuse the edge information of the first image with the sensor data features to form complete first edge information of the first tower crane. The first edge information can accurately express the spatial contour information corresponding to the first tower crane when operating or transporting the first target object.

[0046] In some embodiments, determining the first edge information corresponding to the first tower crane based on the first image data and the first sensor data includes: filtering the first sensor data to obtain corresponding third sensor data; performing image denoising and image edge processing on the first image data to obtain a first image edge; performing image segmentation on the first image data according to the first image edge to obtain a first segmentation result, and performing neighbor region merging processing on the first segmentation result to obtain a first merging result; performing feature change calculation on the first merging result according to the first segmentation result to obtain a first merging quality corresponding to the first merging result; adjusting the first merging result according to the first merging quality to obtain a first target segmentation result; performing target classification according to the first target segmentation result to obtain a first tracking target area corresponding to the first tower crane; and performing data fusion on the first tracking target area and the third sensor data to obtain the first edge information corresponding to the first tower crane under the first sensor.

[0047] Exemplarily, a target filtering algorithm is selected from filtering algorithms such as low-pass filtering, median filtering, and Kalman filtering, and the first sensor data is processed to remove noise, retain valid signals, and generate third sensor data.

[0048] Exemplarily, an image processing algorithm (such as Gaussian filtering, median filtering, etc.) is used to denoise the first image data to reduce noise interference in the image and improve the accuracy of subsequent edge detection, and an edge detection algorithm (such as Canny edge detection, Sobel operator, etc.) is selected to perform edge detection on the denoised image to extract edge information in the image and generate the first image edge.

[0049] Exemplarily, the first image data is segmented according to the edge of the first image by edge-based segmentation, region growth segmentation, etc., and the first image data is segmented into multiple regions to generate a first segmentation result. Then, a region merging algorithm (such as region-based merging, shape-based merging, etc.) is used to merge the neighboring regions in the first segmentation result to generate a first merged result. For example, merging conditions are set, such as region similarity, boundary continuity, etc., to ensure that the merged region has high coherence and consistency.

[0050] Exemplarily, the first color distribution and the first structural distribution before the region merging are calculated based on the first segmentation result and the first merging result, and the second color distribution and the second structural distribution are calculated after the region merging, so as to perform feature change calculation based on the first color distribution and the first structural distribution as well as the second color distribution and the second structural distribution to obtain the first merging quality corresponding to the first merging result.

[0051] Exemplarily, adjustment operations such as region redivision and edge optimization are performed on the first merging result according to the first merging quality to obtain a first target segmentation result.

[0052] Exemplarily, a target classification algorithm (such as feature-based classification, model-based classification, etc.) is determined, and the segmented areas are classified according to the first target segmentation result to determine the first tracking target area corresponding to the first tower crane.

[0053] Exemplarily, the first tracking target area and the third sensor data are aligned in time and space to ensure the consistency of the two in time and space, thereby fusing the first tracking target area and the third sensor data to generate the first edge information of the first tower crane under the first sensor.

[0054] For example, a laser radar is used as the first sensor to obtain point cloud data of the first tower crane in its working environment. The laser radar has the characteristics of high precision and high resolution, and is suitable for high-precision measurement in three-dimensional space. According to the first target segmentation result, the first tracking target area is extracted. The accuracy and integrity of the first target object in the image space are ensured, and then the first tracking target area in the image is matched with the point cloud data to establish a corresponding relationship between the image area and the point cloud data. Methods such as feature point matching and shape matching can be used to ensure the accuracy of the matching. Thus, the first tracking target area is converted from the image coordinate system to the laser radar coordinate system. According to the internal and external parameters of the camera, the coordinates of the target area in the image are converted into the point cloud coordinates of the laser radar. According to the converted coordinates, the corresponding target point cloud data is extracted from the point cloud data. The accuracy and integrity of the target point cloud data are ensured, and the target point cloud data of the first tower crane under the laser radar is generated. Using the target point cloud data, a three-dimensional space model of the first tower crane under the laser radar is constructed. Through three-dimensional reconstruction of the point cloud data, a three-dimensional model of the tower crane and its surrounding environment is generated. Thus, the first edge information of the first tower crane is extracted from the target point cloud data. The first edge information is used to characterize the outline and boundary of the first tower crane in the three-dimensional space.

[0055] In some embodiments, the image denoising and image edge processing of the first image data to obtain a first image edge includes: performing neighborhood calculation on any first point cloud data in the third sensor data to obtain a corresponding first neighborhood set, and calculating a first distance between the first point cloud data and the first neighborhood set; performing mean and standard deviation calculation based on the first distance to obtain a first mean and a first standard deviation, and calculating a first covariance matrix corresponding to the first point cloud data based on the first mean and the first standard deviation; determining a first eigenvector corresponding to the first point cloud data based on the first covariance matrix, and calculating first direction information corresponding to the first point cloud data based on the first neighborhood set; determining a first Gaussian function corresponding to the first image data based on the first eigenvector, the first direction information, the first covariance matrix and the first neighborhood set; performing image denoising on the first image data to obtain a first denoised image, and performing image edge processing on the first denoised image based on the first direction information, the first Gaussian function, the first neighborhood set and the first covariance matrix to obtain the first image edge.

[0056] Exemplarily, a first point cloud data is arbitrarily selected from the third sensor data, and then a neighborhood calculation algorithm (such as K nearest neighbor algorithm, radius neighborhood calculation, etc.) is used to calculate the neighborhood set of the first point cloud data. The neighborhood set includes other point cloud data that are close to the point cloud data in space, so as to obtain a first neighborhood set corresponding to the first point cloud data. Then, the distance between the first point cloud data and each point cloud data in the first neighborhood set is calculated using Euclidean distance or other suitable distance measurement methods to obtain a first distance. Therefore, through neighborhood calculation and distance calculation, the features of the first point cloud data in the local area can be effectively extracted, which is helpful for subsequent feature analysis and processing.

[0057] Exemplarily, according to the first distance between the first point cloud data and the first neighborhood set, a first mean corresponding to the distribution of the first point cloud data is calculated. The first mean represents the overall positional characteristics of the first point cloud data in the first neighborhood set. And according to the first distance between the first point cloud data and the first neighborhood set, a first standard deviation corresponding to the distribution of the first point cloud data is calculated. The first standard deviation represents the degree of dispersion of the first point cloud data in the first neighborhood set.

[0058] Exemplarily, a first covariance matrix corresponding to the first point cloud data is calculated according to the first mean and the first standard deviation. The covariance matrix reflects the distribution characteristics and mutual relationship of any two first point cloud data in the third sensor data calculated according to the first neighborhood set.

[0059] Exemplarily, eigenvalues ​​and eigenvectors are calculated according to the first covariance matrix to obtain a first eigenvector corresponding to the first point cloud data, where the first eigenvector represents a main direction of the first point cloud data in the first neighborhood set.

[0060] Exemplarily, a local plane fitting is performed on the first neighborhood set of each first point cloud data, so that the normal direction of the fitted plane obtained by fitting is determined as the first direction information corresponding to the first point cloud data. The first direction information can represent the orientation or main axis direction of the first point cloud data in space.

[0061] Exemplarily, the first eigenvector and the first direction information are concatenated to form a new vector to obtain a fusion vector, and then the eigenvalue of the fusion vector is adjusted to change the shape of the first covariance matrix. For example, the first covariance matrix is ​​rotated according to the first direction information in the fusion vector to better reflect the directionality of the data. Then, a topological relationship matrix is ​​constructed according to the first neighborhood set to describe the adjacency relationship between the point cloud data: thereby substituting the adjusted first covariance matrix and the topological relationship matrix into the Gaussian function to generate the first Gaussian function, and the first Gaussian function can describe the local distribution characteristics of the first point cloud data.

[0062] Exemplarily, the first image data is subjected to image denoising processing to remove noise in the image. An image denoising algorithm such as Gaussian filtering, median filtering, etc. may be used to obtain the first denoised image.

[0063] Exemplarily, a comprehensive model is designed to fuse the information of the first direction information, the first Gaussian function, the first neighborhood set and the first covariance matrix to integrate the results of different information sources, and establish a probability model to comprehensively consider the probability distribution of different information sources. For example, a Bayesian network can be constructed to combine the probability distribution of different information sources, and then the edge response of each pixel is calculated based on the fusion result. For example, the edge response can be calculated by comparing the intensity change of the local area. According to the threshold of the edge response, the edge position in the first image data is located, so that the detected edge position generates the first image edge.

[0064] Specifically, through neighborhood calculation and distance calculation, the features of point cloud data in local areas can be effectively extracted, providing a basis for subsequent feature analysis and processing. Through the calculation of mean and standard deviation, the distribution of point cloud data in the neighborhood set can be analyzed, providing a basis for the subsequent calculation of feature vectors and direction information. Through the calculation of feature vectors and direction information, the orientation and main axis direction features of point cloud data can be extracted, which is helpful for subsequent image edge processing and Gaussian function generation. Through Gaussian function and image denoising processing, point cloud data and image data can be smoothed, noise interference can be removed, and data quality and reliability can be improved, thereby ensuring the accuracy, stability and reliability of point cloud data and image data during the processing process, and providing strong data support for subsequent anti-collision control and tower crane collaborative operation.

[0065] Step S104: Determine second edge information corresponding to the second tower crane according to the second image data and the second sensor data.

[0066] Exemplarily, a suitable edge detection algorithm is selected, such as Canny edge detection, Sobel operator, Laplacian operator, etc. Parameters of the edge detection algorithm are set according to the second image data, such as high and low thresholds, convolution kernel size, etc. Ensure that the algorithm can detect the second image edge information corresponding to the second target object in the second image data. And refine the detected edges, remove redundant information, and enhance the clarity and continuity of the edges. Morphological processing (such as dilation, corrosion) and other techniques can be used to optimize the edges.

[0067] Exemplarily, feature information related to the edge information of the second image is extracted from the second sensing data, and then the extracted sensing data features are matched with the edge information of the second image to ensure the consistency and relevance of the information between the two, thereby fusing the edge information of the second image with the sensing data features to form complete second edge information of the second tower crane. The second edge information can accurately express the spatial contour information corresponding to the second tower crane when operating or transporting the second target object.

[0068] In some embodiments, determining the second edge information corresponding to the second tower crane based on the second image data and the second sensor data includes: filtering the second sensor data to obtain corresponding fourth sensor data; performing image denoising and image edge processing on the second image data to obtain a second image edge; performing image segmentation on the second image data according to the second image edge to obtain a second segmentation result, and performing neighbor region merging processing on the second segmentation result to obtain a second merging result; performing feature change calculation on the second merging result according to the second segmentation result to obtain a second merging quality corresponding to the second merging result; adjusting the second merging result according to the second merging quality to obtain a second target segmentation result; performing target classification according to the second target segmentation result to obtain a second tracking target area corresponding to the second tower crane; and performing data fusion on the second tracking target area and the fourth sensor data to obtain the second edge information corresponding to the second tower crane under the second sensor.

[0069] Exemplarily, a target filtering algorithm is selected from filtering algorithms such as low-pass filtering, median filtering, and Kalman filtering, and the second sensor data is processed to remove noise, retain valid signals, and generate fourth sensor data.

[0070] Exemplarily, an image processing algorithm (such as Gaussian filtering, median filtering, etc.) is used to denoise the second image data to reduce noise interference in the image and improve the accuracy of subsequent edge detection, and an edge detection algorithm (such as Canny edge detection, Sobel operator, etc.) is selected to perform edge detection on the denoised image to extract edge information in the image and generate the second image edge.

[0071] Exemplarily, the second image data is segmented according to the edge of the second image by edge-based segmentation, region growth segmentation, etc., and the second image data is segmented into multiple regions to generate a second segmentation result. Then, a region merging algorithm (such as region-based merging, shape-based merging, etc.) is used to merge the neighboring regions in the second segmentation result to generate a second merged result. For example, merging conditions are set, such as region similarity, boundary continuity, etc., to ensure that the merged region has high coherence and consistency.

[0072] Exemplarily, the third color distribution and the third structure distribution before the region merging are calculated based on the second segmentation result and the second merging result, and the fourth color distribution and the fourth structure distribution after the region merging are calculated, so that the feature change calculation is performed based on the third color distribution and the third structure distribution as well as the fourth color distribution and the fourth structure distribution to obtain the second merging quality corresponding to the second merging result.

[0073] Exemplarily, adjustment operations such as region redivision and edge optimization are performed on the second merging result according to the second merging quality to obtain a second target segmentation result.

[0074] Exemplarily, a target classification algorithm (such as feature-based classification, model-based classification, etc.) is determined, and the segmented areas are classified according to the second target segmentation result to determine the second tracking target area corresponding to the second tower crane.

[0075] Exemplarily, the second tracking target area and the fourth sensor data are aligned in time and space to ensure their consistency in time and space, thereby fusing the second tracking target area and the fourth sensor data to generate second edge information of the second tower crane under the second sensor.

[0076] For example, a laser radar is used as a second sensor to obtain point cloud data of the second tower crane in its working environment. The laser radar has the characteristics of high precision and high resolution, and is suitable for high-precision measurement in three-dimensional space. According to the second target segmentation result, the second tracking target area is extracted. The accuracy and completeness of the second target object in the image space are ensured, and then the second tracking target area in the image is matched with the point cloud data to establish a corresponding relationship between the image area and the point cloud data. Methods such as feature point matching and shape matching can be used to ensure the accuracy of the matching. Thus, the second tracking target area is converted from the image coordinate system to the laser radar coordinate system. According to the internal and external parameters of the camera, the coordinates of the target area in the image are converted into the point cloud coordinates of the laser radar. According to the converted coordinates, the corresponding target point cloud data is extracted from the point cloud data. The accuracy and completeness of the target point cloud data are ensured, and the target point cloud data of the second tower crane under the laser radar is generated. Using the target point cloud data, a three-dimensional space model of the second tower crane under the laser radar is constructed. Through three-dimensional reconstruction of the point cloud data, a three-dimensional model of the tower crane and its surrounding environment is generated. Thus, the second edge information of the second tower crane is extracted from the target point cloud data. The second edge information is used to characterize the outline and boundary of the second tower crane in the three-dimensional space.

[0077] In some embodiments, the image denoising and image edge processing of the second image data to obtain a second image edge includes: performing neighborhood calculation on any second point cloud data in the fourth sensor data to obtain a corresponding second neighborhood set, and calculating a second distance between the second point cloud data and the second neighborhood set; performing mean and standard deviation calculation based on the second distance to obtain a second mean and a second standard deviation, and calculating a second covariance matrix corresponding to the second point cloud data based on the second mean and the second standard deviation; determining a second eigenvector corresponding to the second point cloud data based on the second covariance matrix, and calculating second direction information corresponding to the second point cloud data based on the second neighborhood set; determining a second Gaussian function corresponding to the second image data based on the second eigenvector, the second direction information, the second covariance matrix and the second neighborhood set; performing image denoising on the second image data to obtain a second denoised image, and performing image edge processing on the second denoised image based on the second direction information, the second Gaussian function, the second neighborhood set and the second covariance matrix to obtain the second image edge.

[0078] Exemplarily, a second point cloud data is arbitrarily selected from the fourth sensor data, and then a neighborhood calculation algorithm (such as K nearest neighbor algorithm, radius neighborhood calculation, etc.) is used to calculate the neighborhood set of the second point cloud data. The neighborhood set includes other point cloud data that are spatially close to the point cloud data, thereby obtaining a second neighborhood set corresponding to the second point cloud data. Then, the distance between the second point cloud data and each point cloud data in the second neighborhood set is calculated using Euclidean distance or other suitable distance measurement methods to obtain a second distance. Therefore, through neighborhood calculation and distance calculation, the features of the second point cloud data in the local area can be effectively extracted, which is helpful for subsequent feature analysis and processing.

[0079] Exemplarily, according to the second distance between the second point cloud data and the second neighborhood set, the second mean corresponding to the distribution of the second point cloud data is calculated. The second mean represents the overall positional characteristics of the second point cloud data in the second neighborhood set. And according to the second distance between the second point cloud data and the second neighborhood set, the second standard deviation corresponding to the distribution of the second point cloud data is calculated. The second standard deviation represents the degree of dispersion of the second point cloud data in the second neighborhood set.

[0080] Exemplarily, a second covariance matrix corresponding to the second point cloud data is calculated based on the second mean and the second standard deviation. The second covariance matrix reflects the distribution characteristics and mutual relationships of any two second point cloud data in the fourth sensor data calculated based on the second neighborhood set.

[0081] Exemplarily, eigenvalue and eigenvector calculations are performed according to the second covariance matrix to obtain a second eigenvector corresponding to the second point cloud data, where the second eigenvector represents a main direction of the second point cloud data in the second neighborhood set.

[0082] Exemplarily, a local plane fitting is performed on the second neighborhood set of each second point cloud data, so that the normal direction of the fitted plane obtained by fitting is determined as the second direction information corresponding to the second point cloud data. The second direction information can represent the orientation or main axis direction of the second point cloud data in space.

[0083] Exemplarily, the second eigenvector and the second direction information are concatenated to form a new vector to obtain a fusion vector, and then the eigenvalue of the fusion vector is adjusted to change the shape of the second covariance matrix. For example, the second covariance matrix is ​​rotated according to the second direction information in the fusion vector to better reflect the directionality of the data. Then, a topological relationship matrix is ​​constructed according to the second neighborhood set to describe the adjacency relationship between the point cloud data: thereby substituting the adjusted second covariance matrix and the topological relationship matrix into the Gaussian function to generate a second Gaussian function, which can describe the local distribution characteristics of the second point cloud data.

[0084] Exemplarily, the second image data is subjected to image denoising to remove noise in the image. An image denoising algorithm such as Gaussian filtering or median filtering may be used to obtain the second denoised image.

[0085] Exemplarily, a comprehensive model is designed to fuse the second direction information, the second Gaussian function, the second neighborhood set, and the second covariance matrix information to synthesize the results of different information sources, and establish a probability model to comprehensively consider the probability distribution of different information sources. For example, a Bayesian network can be constructed to combine the probability distributions of different information sources, and then the edge response of each pixel is calculated based on the fusion result. For example, the edge response can be calculated by comparing the intensity changes in the local area. According to the threshold of the edge response, the edge position in the second image data is located, so that the detected edge position generates the second image edge.

[0086] Specifically, through neighborhood calculation and distance calculation, the features of point cloud data in local areas can be effectively extracted, providing a basis for subsequent feature analysis and processing. Through the calculation of mean and standard deviation, the distribution of point cloud data in the neighborhood set can be analyzed, providing a basis for the subsequent calculation of feature vectors and direction information. Through the calculation of feature vectors and direction information, the orientation and main axis direction features of point cloud data can be extracted, which is helpful for subsequent image edge processing and Gaussian function generation. Through Gaussian function and image denoising processing, point cloud data and image data can be smoothed, noise interference can be removed, and data quality and reliability can be improved, thereby ensuring the accuracy, stability and reliability of point cloud data and image data during the processing process, and providing strong data support for subsequent anti-collision control and tower crane collaborative operation.

[0087] Step S105: using the Beidou positioning system to obtain first position information corresponding to the first tower crane and using the Beidou positioning system to obtain second position information corresponding to the second tower crane.

[0088] Exemplarily, the first sensor of the Beidou positioning system installed on the first tower crane can capture the first target position corresponding to the relevant information, so as to obtain the first position information corresponding to the first tower crane using the first target position according to the Beidou positioning system. For example, the first target position is the tower arm position of the first tower crane.

[0089] For example, the second sensor of the Beidou positioning system installed on the second tower crane can capture the second target position corresponding to the relevant information, so as to obtain the second position information corresponding to the second tower crane using the second target position according to the Beidou positioning system. For example, the second target position is the tower arm position of the second tower crane.

[0090] Step S106: Determine first boundary information corresponding to the first tower crane according to the first edge information and the first position information.

[0091] Exemplarily, the first edge information is used to characterize the outline and boundary of the first tower crane in the three-dimensional space. The first position information is used to characterize the position information corresponding to the first component of the Beidou positioning system installed in the first tower crane. Thus, the third position information corresponding to the first component in the first sensing data is identified, and the third position information is converted to the first predicted real position corresponding to the same coordinate system as the Beidou positioning system, and then the first predicted real position and the first position information are subjected to error calculation to obtain the first error, and then the first edge information is subjected to data correction according to the first error to obtain the first corrected edge information, thereby obtaining the minimum circumscribed matrix corresponding to the first corrected edge information, thereby obtaining the first boundary information corresponding to the first tower crane.

[0092] In some embodiments, determining the first boundary information corresponding to the first tower crane based on the first edge information and the first position information includes: obtaining the first installation position of the Beidou positioning system corresponding to the first tower crane, and determining the first target mark corresponding to the first installation position; tracking the first target mark according to the first image data and the first sensor data to obtain the third edge information corresponding to the first target mark; calculating the center point according to the third edge information to obtain the first target center, and determining the first coordinate information corresponding to the first target center; calculating the first position difference between the first coordinate information and the first position information, and performing data correction on the first edge information according to the first position difference to obtain the first boundary information corresponding to the first tower crane.

[0093] Exemplarily, the first installation position is a position corresponding to the first component of the Beidou positioning system on the first tower crane, and the first target mark for tracking the Beidou positioning system at the first installation position on the first tower crane is determined. That is, the first component of the Beidou positioning system installed on the first tower crane can be tracked through the first target mark.

[0094] Exemplarily, a filter (such as Gaussian filtering, median filtering) is used to remove noise in the first image data, and the contrast of the first image data is adjusted to make the first target mark in the first image data more obvious. Thus, the pre-prepared target mark template is matched with the first image data to find the approximate position of the target mark. The key feature points of the first target mark are extracted from the first image data using a feature extraction algorithm (such as SIFT, SURF, ORB). The extracted feature points are matched with the target mark template using a matching algorithm (such as FLANN matcher, BFMatcher), the position information of the first target mark in the first image data is determined, and then the position information of the first target mark in the first image data is fused with the first sensor data to obtain more accurate third edge information of the first target mark in three-dimensional space. That is, the third edge information is the contour information of the first target mark in three-dimensional space.

[0095] Exemplarily, the third edge information is processed to extract the outline of the first target mark, and then the center point of the first target mark is calculated using a geometric method (such as the centroid method, the center point of the minimum enclosing rectangle, etc.) to obtain the first target center. The first target center is then mapped to the world coordinate system corresponding to the Beidou positioning system to determine the first coordinate information corresponding to the first target center.

[0096] Exemplarily, the first position information is the position information obtained by the Beidou coordinate system to accurately locate the first target mark, the first coordinate information is the position information identified after target tracking based on the first image data and the first sensor data, and then the first position difference between the first coordinate information and the first position information is calculated, the first position difference is the positioning error identified by target tracking based on the first image data and the first sensor data, so that the first edge information is converted to the world coordinate system corresponding to the Beidou positioning system and then corrected using the first position difference to obtain the real position information corresponding to the fourth edge information in the world coordinate system, and finally, the minimum circumscribed matrix is ​​solved for the real position information corresponding to the fourth edge information in the world coordinate system to obtain the first boundary information corresponding to the first tower crane.

[0097] Step S107: Determine second boundary information corresponding to the second tower crane according to the second edge information and the second position information.

[0098] Exemplarily, the second edge information is used to characterize the outline and boundary of the second tower crane in the three-dimensional space. The second position information is used to characterize the corresponding position information of the second component of the Beidou positioning system installed in the second tower crane. Thus, the fourth position information corresponding to the second component in the second sensing data is identified, and the fourth position information is converted to the corresponding second predicted real position in the same coordinate system as the Beidou positioning system, and then the second predicted real position and the second position information are subjected to error calculation to obtain the second error, and then the second edge information is corrected according to the second error to obtain the second corrected edge information, thereby obtaining the minimum circumscribed matrix corresponding to the second corrected edge information, thereby obtaining the second boundary information corresponding to the second tower crane.

[0099] In some embodiments, determining the second boundary information corresponding to the second tower crane based on the second edge information and the second position information includes: obtaining the second installation position of the Beidou positioning system corresponding to the second tower crane, and determining the second target mark corresponding to the second installation position; tracking the second target mark according to the second image data and the second sensor data to obtain fourth edge information corresponding to the second target mark; calculating the center point according to the fourth edge information to obtain the second target center, and determining the second coordinate information corresponding to the second target center; calculating the second position difference between the second coordinate information and the second position information, and performing data correction on the second edge information according to the second position difference to obtain the second boundary information corresponding to the second tower crane.

[0100] Exemplarily, the second installation position is a position corresponding to the Beidou positioning system on the second component of the second tower crane, and the second installation position is determined to be a second target mark on the second tower crane for tracking the Beidou positioning system. That is, the second component of the Beidou positioning system installed on the second tower crane can be tracked through the second target mark.

[0101] Exemplarily, a filter (such as Gaussian filtering, median filtering) is used to remove noise in the second image data, and the contrast of the second image data is adjusted to make the second target mark in the second image data more obvious. Thus, the pre-prepared target mark template is matched with the second image data to find the approximate position of the target mark. The key feature points of the second target mark are extracted from the second image data using a feature extraction algorithm (such as SIFT, SURF, ORB). The extracted feature points are matched with the target mark template using a matching algorithm (such as FLANN matcher, BFMatcher), and the position information of the second target mark in the second image data is determined, and then the position information of the second target mark in the second image data is fused with the second sensor data to obtain more accurate fourth edge information of the second target mark in three-dimensional space. That is, the fourth edge information is the contour information of the second target mark in three-dimensional space.

[0102] Exemplarily, the fourth edge information is processed to extract the outline of the second target mark, and then the center point of the second target mark is calculated using a geometric method (such as the centroid method, the center point of the minimum enclosing rectangle, etc.) to obtain the second target center. The second target center is then mapped to the world coordinate system corresponding to the Beidou positioning system to determine the second coordinate information corresponding to the second target center.

[0103] Exemplarily, the second position information is the position information obtained by accurately locating the second target mark by the Beidou coordinate system, and the second coordinate information is the position information identified after target tracking based on the second image data and the second sensor data, and then the second position difference between the second coordinate information and the second position information is calculated, and the second position difference is the positioning error identified by target tracking based on the second image data and the second sensor data, so that the fourth edge information is converted to the world coordinate system corresponding to the Beidou positioning system and then corrected using the second position difference to obtain the real position information corresponding to the fourth edge information in the world coordinate system, and finally, the minimum circumscribed matrix is ​​solved for the real position information corresponding to the fourth edge information in the world coordinate system to obtain the second boundary information corresponding to the second tower crane.

[0104] Step S108: determining a tower group operation rule, and determining a target operation state corresponding to the first tower crane and the second tower crane according to the tower group operation rule combined with the first boundary information and the second boundary information.

[0105] Exemplarily, the tower group operation rules are determined based on expert experience or historical experience. The tower group operation rules refer to the corresponding operation rules when multiple tower cranes work together. For example, the tower group operation rules are "the rear tower gives way to the front tower", "the moving tower gives way to the static tower", "the light vehicle gives way to the heavy vehicle", "the guest tower gives way to the main tower" and other principles of giving way, ensuring that the horizontal distance between the end of the low tower boom and the high tower body is not less than 5 meters, and the vertical clearance between the high tower boom and hook and the highest point of the low tower is not less than 2 meters. When the safety distance between tower cranes is less than the preset value, the system automatically sends out a warning signal, and can intervene in the operation of the tower crane, adjust the speed or stop the action to prevent collision accidents.

[0106] Exemplarily, based on the comparison between the first boundary information and the second boundary information, when the horizontal distance between the first tower crane and the second tower crane is less than the first preset value, the target operating state corresponding to the first tower crane and the second tower crane is determined to be abnormal operation, and when the vertical distance between the first tower crane and the second tower crane is less than the second preset value, the target operating state corresponding to the first tower crane and the second tower crane is determined to be abnormal operation; when the horizontal distance between the first tower crane and the second tower crane is greater than or equal to the first preset value and the vertical distance between the first tower crane and the second tower crane is greater than or equal to the second preset value, the target operating state corresponding to the first tower crane and the second tower crane is determined to be normal operation.

[0107] Step S109: performing anti-collision adjustment on the first tower crane and the second tower crane according to the target operating state to obtain target processing results of the first tower crane and the second tower crane.

[0108] Exemplarily, according to the comparison between the first boundary information and the second boundary information, when the horizontal distance between the first tower crane and the second tower crane is greater than or equal to the first preset value, and the vertical distance between the first tower crane and the second tower crane is greater than or equal to the second preset value, it is determined that the target operation state corresponding to the first tower crane and the second tower crane is normal operation, and the first tower crane and the second tower crane can work normally; when the horizontal distance between the first tower crane and the second tower crane is less than the first preset value, or when the vertical distance between the first tower crane and the second tower crane is less than the second preset value, it is determined that the target operation state corresponding to the first tower crane and the second tower crane is abnormal operation; then it is necessary to intervene in the first tower crane and the second tower crane, including but not limited to adjusting the speed or stopping the action. When intervening in the first tower crane and the second tower crane, it is also necessary to comply with the group tower operation rules such as "the rear tower gives way to the front tower", "the moving tower gives way to the static tower", "the light vehicle gives way to the heavy vehicle", "the passenger tower gives way to the main tower" and other giving way principles.

[0109] Exemplarily, when the target operating state is abnormal operation, the first tower crane and the second tower crane are adjusted according to the tower group operating rules such as "the rear tower gives way to the front tower", "the moving tower gives way to the static tower", "the light vehicle gives way to the heavy vehicle", and "the passenger tower gives way to the main tower" to obtain the target processing result.

[0110] For example, whether the first tower crane is a moving tower or a static tower is determined according to whether the first edge information of the first tower crane changes at different times, and the weight information of the first operation object corresponding to the first tower crane and the weight information of the second operation object corresponding to the second tower crane are obtained, thereby determining whether the first tower crane is a light vehicle or a heavy vehicle, and determining whether the first tower crane is a rear tower or a front tower according to the first boundary information and the second boundary information, etc., thereby performing anti-collision adjustments on the first tower crane and the second tower crane according to the tower group operation rules and the target operation status to obtain the target processing results of the first tower crane and the second tower crane.

[0111] In some embodiments, determining the tower group operation rules includes: obtaining a tower group operation text, and performing relationship extraction on the tower group operation text to obtain initial rules corresponding to the tower group operation text; performing priority identification on the initial rules according to the tower group operation text to obtain target priorities corresponding to the initial rules; and adjusting the initial rules according to the target priorities to obtain the tower group operation rules.

[0112] Exemplarily, the tower cluster operation text is obtained from relevant documents, manuals, operation guides and other resources, and the obtained tower cluster operation text is cleaned to remove irrelevant information and noise to ensure the purity of the text data, and then the tower cluster operation text is analyzed using natural language processing to identify the relationships and rules in the tower cluster operation text. For example, dependency syntax analysis, named entity recognition and other technologies can be used to extract initial rules from the tower cluster operation text.

[0113] Exemplarily, the priority of the initial rules is identified according to the tower group operation text to obtain the target priority, for example, the keywords and key phrases in the tower group operation text are identified, and these keywords and phrases are usually related to the priority of the rules. For example, words such as "must", "priority", "urgent", etc., or the priority weight of each initial rule is calculated according to the results of keyword and context analysis. For example, a weighted scoring method can be used to weight the weights of keywords and contexts.

[0114] Exemplarily, the initial rules are adjusted according to the target priority to obtain the tower group operation rules, and the initial rules are sorted according to the target priority to ensure that the high-priority rules are executed first. Rules with the same or similar priorities are merged to simplify the rule set. For example, multiple rules with similar conditions can be merged into one rule, etc., to obtain the tower group operation rules.

[0115] In some embodiments, the performing of relationship extraction on the tower group operation text to obtain the initial rules corresponding to the tower group operation text includes: performing entity representation on the tower group operation text using the entity representation layer of the relationship extraction model to obtain a first entity representation vector corresponding to the first entity and a second entity representation vector corresponding to the second entity in the tower group operation text; performing relationship information extraction on the first entity and the second entity using the entity information extraction layer of the relationship extraction model to obtain the relationship representation vector corresponding to the first entity and the second entity; performing similarity calculation on the first entity representation vector and the relationship representation vector using the similarity calculation layer of the relationship extraction model to obtain a first similarity between the first entity representation vector and the relationship representation vector, and performing similarity calculation on the second entity representation vector and the relationship representation vector to obtain a second similarity between the second entity representation vector and the relationship representation vector; performing similarity calculation on the second entity representation vector and the relationship representation vector using the relationship extraction model to obtain a first similarity between the first entity representation vector and the relationship representation vector; performing similarity calculation on the second entity representation vector and the relationship representation vector to obtain a second similarity between the second entity representation vector and the relationship representation vector; performing similarity calculation on the second entity representation vector and the relationship representation vector using the relationship extraction model to obtain a first similarity between the first entity representation vector and the relationship representation vector; performing similarity calculation on the second entity representation vector and the relationship representation vector The context feature extraction layer of the model extracts the context relationship of the first entity to obtain a first context representation vector and extracts the context relationship of the second entity to obtain a second context representation vector; the relationship feature extraction layer of the relationship extraction model determines the first features corresponding to the first entity according to the first similarity, the second similarity, the first entity representation vector and the first context representation vector, and determines the second features corresponding to the second entity according to the first similarity, the second similarity, the second entity representation vector and the second context representation vector; the relationship classification layer of the relationship extraction model determines the target entity relationship between the first entity and the second entity according to the first feature and the second feature; the initial rule corresponding to the group tower operation text is determined according to the first entity, the second entity and the target entity relationship; wherein the first feature and the second feature are obtained according to the following formula:

[0116] x i =NN([ e i , e re , ∑ t=1 m softmax( w t * c t )* c t ]);

[0117] x j =NN([ e j , e re , ∑ k=1 n softmax( w k * c k )* c k ]);

[0118]

[0119] in, represents the first feature corresponding to the i-th first entity, represents the second feature corresponding to the j-th second entity; represents the first entity representation vector corresponding to the i-th first entity, represents the second entity representation vector corresponding to the j-th second entity, represents an entity relationship vector between the first entity and the second entity, represents a first similarity between the first entity representation vector corresponding to the i-th first entity and the relationship representation vector; represents the second similarity between the second entity representation vector corresponding to the j-th second entity and the relationship representation vector, m represents the first number of associations between the first entity and other entities, represents the tth first context representation vector, n represents the second number of associations between the second entity and other entities, represents the kth second context representation vector, NN represents the neural network corresponding to the relationship feature extraction layer, Indicates that when the first similarity Greater than the second similarity hour, Equal to the first entity representation vector , when the first similarity Less than or equal to the second similarity hour, Equal to the second entity representation vector .

[0120] Exemplarily, the relationship extraction model is a machine learning model used to extract entities and their relationships from text, and generally includes an entity representation layer, an entity information extraction layer, a similarity calculation layer, a context feature extraction layer, a relationship feature extraction layer, and a relationship classification layer.

[0121] Exemplarily, the entity representation layer of the relational extraction model is used to perform entity representation on the tower group operation text, and the first entity representation vector corresponding to the first entity and the second entity representation vector corresponding to the second entity in the tower group operation text are obtained. For example, the named entity recognition technology is used to identify the first entity and the second entity from the tower group operation text. For example, the "front tower crane" and "back tower crane" are identified, so that the recognized entities are mapped to the representation form in the high-dimensional vector space, which can capture the semantic information of the entity. For example, the entity is mapped to a representation vector using a pre-trained word embedding model (such as Word2Vec, GloVe, BERT). For example, the first entity representation vector corresponding to the "front tower crane" and the second entity representation vector corresponding to the "back tower crane" are obtained.

[0122] Exemplarily, the entity information extraction layer of the relationship extraction model is used to extract relationship information from the first entity and the second entity to obtain a relationship representation vector.

[0123] Exemplarily, the similarity calculation layer of the relationship extraction model is used to calculate the first similarity between the first entity representation vector and the relationship representation vector by using methods such as cosine similarity and Euclidean distance, and the second similarity between the second entity representation vector and the relationship representation vector is calculated by using methods such as cosine similarity and Euclidean distance.

[0124] Exemplarily, the context feature extraction layer of the relation extraction model is used to extract contextual relations, analyze the text context information around the first entity and the second entity, and extract the context features related to them. For example, the context of "the tower crane in front" and "the tower crane behind" in the text of the tower group operation is analyzed, and the operation steps, safety requirements, etc. related to them are extracted, so as to obtain the first context representation vector corresponding to the first entity and the second context representation vector corresponding to the second entity.

[0125] Exemplarily, the relational feature extraction layer of the relational extraction model is used to fuse the first entity representation vector, the first similarity, the second similarity, and the first context representation vector according to the following formula to generate the first feature of the first entity. Similarly, the second entity representation vector, the second similarity, the first similarity, and the second context representation vector are fused according to the following formula to generate the second feature of the second entity. The first feature and the second feature are obtained according to the following formula:

[0126] x i =NN([ e i , e re , ∑ t=1 m softmax( w t * c t )* c t ]);

[0127] x j =NN([ e j , e re , ∑ k=1 n softmax( w k * c k )* c k ]);

[0128]

[0129] in, represents the first feature corresponding to the i-th first entity, represents the second feature corresponding to the j-th second entity; represents the first entity representation vector corresponding to the i-th first entity, represents the second entity representation vector corresponding to the j-th second entity, represents the entity relationship vector between the first entity and the second entity, Represents the first similarity between the first entity representation vector and the relationship representation vector corresponding to the i-th first entity; represents the second similarity between the second entity representation vector and the relationship representation vector corresponding to the j-th second entity, m represents the first number of associations between the first entity and other entities, represents the tth first context representation vector, n represents the second number of associations between the second entity and other entities, represents the kth second context representation vector, and NN represents the neural network corresponding to the relational feature extraction layer.

[0130] Exemplarily, the relationship between the first entity and the second entity is classified by using a classification algorithm (such as logistic regression, support vector machine, neural network) based on the first feature and the second feature using the relationship classification layer of the relationship extraction model. The classification result is the target entity relationship. For example, the classification result is priority or no priority. Thus, according to the first entity, the second entity and the target entity relationship, the initial rule corresponding to the group tower operation text is determined. For example, the first entity, the second entity and the target entity relationship are combined to generate an initial rule. For example, the rule "front tower crane - rear tower crane - no priority" is generated. That is, the rear tower crane takes precedence over the front tower crane.

[0131] See also Figure 2 , Figure 2A group-tower intelligent collaborative anti-collision system 200 is provided in an embodiment of the present application. The group-tower intelligent collaborative anti-collision system 200 includes a first acquisition module 201, a second acquisition module 202, a first processing module 203, a second processing module 204, a position determination module 205, a third processing module 206, a fourth processing module 207, a state determination module 208, and a data adjustment module 209, wherein the first acquisition module 201 is used to use a first camera to acquire first image data corresponding to a first tower crane and use a first sensor to acquire first sensor data corresponding to the first tower crane; the second acquisition module 202 is used to use a second camera to acquire second image data corresponding to a second tower crane and use a second sensor to acquire second sensor data corresponding to the second tower crane; the first processing module 203 is used to determine first edge information corresponding to the first tower crane according to the first image data and the first sensor data; the second processing module 204 is used to determine first edge information corresponding to the first tower crane according to the second image data and the first sensor data The second edge information corresponding to the second tower crane is determined by the second sensor data; a position determination module 205 is used to obtain the first position information corresponding to the first tower crane by using the Beidou positioning system and to obtain the second position information corresponding to the second tower crane by using the Beidou positioning system; a third processing module 206 is used to determine the first boundary information corresponding to the first tower crane according to the first edge information and the first position information; a fourth processing module 207 is used to determine the second boundary information corresponding to the second tower crane according to the second edge information and the second position information; a state determination module 208 is used to determine the tower group operation rules, and determine the target operation states corresponding to the first tower crane and the second tower crane according to the tower group operation rules combined with the first boundary information and the second boundary information; a data adjustment module 209 is used to perform anti-collision adjustments on the first tower crane and the second tower crane according to the target operation state, and obtain the target processing results of the first tower crane and the second tower crane.

[0132] In some implementations, the tower cluster intelligent collaborative collision avoidance system 200 may be applied to terminal devices.

[0133] It should be noted that, those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described tower group intelligent collaborative collision avoidance system 200 can refer to the corresponding process in the aforementioned tower group intelligent collaborative collision avoidance method embodiment, and will not be repeated here.

[0134] An embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of the methods for intelligent collaborative collision avoidance of tower groups provided in the description of the embodiments of the present invention.

[0135] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0136] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0137] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0138] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for intelligent collaborative anti-collision of tower groups, characterized in that: The method comprises: Using a first camera to collect first image data corresponding to a first tower crane and using a first sensor to collect first sensor data corresponding to the first tower crane; Using a second camera to collect second image data corresponding to a second tower crane and using a second sensor to collect second sensor data corresponding to the second tower crane; Determine first edge information corresponding to the first tower crane according to the first image data and the first sensor data; Determine second edge information corresponding to the second tower crane according to the second image data and the second sensor data; Using the Beidou positioning system to obtain first position information corresponding to the first tower crane and using the Beidou positioning system to obtain second position information corresponding to the second tower crane; Determine first boundary information corresponding to the first tower crane according to the first edge information and the first position information; Determine second boundary information corresponding to the second tower crane according to the second edge information and the second position information; Determine a tower group operation rule, and determine a target operation state corresponding to the first tower crane and the second tower crane according to the tower group operation rule combined with the first boundary information and the second boundary information; Anti-collision adjustments are performed on the first tower crane and the second tower crane according to the target operating state to obtain target processing results of the first tower crane and the second tower crane.

2. The method according to claim 1, characterized in that The determining, according to the first image data and the first sensor data, first edge information corresponding to the first tower crane comprises: Performing filtering on the first sensing data to obtain corresponding third sensing data; Performing image denoising and image edge processing on the first image data to obtain a first image edge; Performing image segmentation on the first image data according to the first image edge to obtain a first segmentation result, and performing neighboring region merging processing on the first segmentation result to obtain a first merging result; Performing feature change calculation on the first merging result according to the first segmentation result to obtain a first merging quality corresponding to the first merging result; Adjusting the first merging result according to the first merging quality to obtain a first target segmentation result; Performing target classification according to the first target segmentation result to obtain a first tracking target area corresponding to the first tower crane; The first tracking target area and the third sensor data are data-fused to obtain the first edge information corresponding to the first tower crane under the first sensor.

3. The method according to claim 2, characterized in that The performing image denoising and image edge processing on the first image data to obtain a first image edge includes: Performing neighborhood calculation on any one of the first point cloud data in the third sensing data to obtain a corresponding first neighborhood set, and calculating a first distance between the first point cloud data and the first neighborhood set; Calculating a first mean and a first standard deviation based on the first distance, and calculating a first covariance matrix corresponding to the first point cloud data based on the first mean and the first standard deviation; Determine a first eigenvector corresponding to the first point cloud data according to the first covariance matrix, and calculate first direction information corresponding to the first point cloud data according to the first neighborhood set; Determine a first Gaussian function corresponding to the first image data according to the first eigenvector, the first direction information, the first covariance matrix, and the first neighborhood set; Performing image denoising on the first image data to obtain a first denoised image, and performing image edge processing on the first denoised image according to the first direction information, the first Gaussian function, the first neighborhood set and the first covariance matrix to obtain the first image edge.

4. The method according to claim 1, characterized in that The determining the second edge information corresponding to the second tower crane according to the second image data and the second sensor data includes: Performing filtering processing on the second sensing data to obtain corresponding fourth sensing data; Performing image denoising and image edge processing on the second image data to obtain a second image edge; Performing image segmentation on the second image data according to the second image edge to obtain a second segmentation result, and performing neighboring region merging processing on the second segmentation result to obtain a second merging result; Performing feature change calculation on the second merging result according to the second segmentation result to obtain a second merging quality corresponding to the second merging result; Adjusting the second merging result according to the second merging quality to obtain a second target segmentation result; Performing target classification according to the second target segmentation result to obtain a second tracking target area corresponding to the second tower crane; The second tracking target area and the fourth sensor data are data-fused to obtain the second edge information corresponding to the second tower crane under the second sensor.

5. The method according to claim 4, characterized in that The performing image denoising and image edge processing on the second image data to obtain a second image edge includes: Performing neighborhood calculation on any second point cloud data in the fourth sensing data to obtain a corresponding second neighborhood set, and calculating a second distance between the second point cloud data and the second neighborhood set; Calculating the mean and the standard deviation according to the second distance to obtain a second mean and a second standard deviation, and calculating a second covariance matrix corresponding to the second point cloud data according to the second mean and the second standard deviation; Determine a second eigenvector corresponding to the second point cloud data according to the second covariance matrix, and calculate second direction information corresponding to the second point cloud data according to the second neighborhood set; Determine a second Gaussian function corresponding to the second image data according to the second eigenvector, the second direction information, the second covariance matrix, and the second neighborhood set; Perform image denoising on the second image data to obtain a second denoised image, and perform image edge processing on the second denoised image according to the second direction information, the second Gaussian function, the second neighborhood set and the second covariance matrix to obtain the second image edge.

6. The method according to claim 1, characterized in that The determining, according to the first edge information and the first position information, first boundary information corresponding to the first tower crane includes: Obtaining a first installation position of the Beidou positioning system corresponding to the first tower crane, and determining a first target mark corresponding to the first installation position; Tracking the first target mark according to the first image data and the first sensing data to obtain third edge information corresponding to the first target mark; Performing center point calculation according to the third edge information to obtain a first target center, and determining first coordinate information corresponding to the first target center; A first position difference between the first coordinate information and the first position information is calculated, and data correction is performed on the first edge information according to the first position difference to obtain the first boundary information corresponding to the first tower crane.

7. The method according to claim 1, characterized in that The determining, according to the second edge information and the second position information, second boundary information corresponding to the second tower crane comprises: Obtaining a second installation position of the Beidou positioning system corresponding to the second tower crane, and determining a second target mark corresponding to the second installation position; Tracking the second target mark according to the second image data and the second sensing data to obtain fourth edge information corresponding to the second target mark; Performing center point calculation according to the fourth edge information to obtain a second target center, and determining second coordinate information corresponding to the second target center; A second position difference between the second coordinate information and the second position information is calculated, and data correction is performed on the second edge information according to the second position difference to obtain the second boundary information corresponding to the second tower crane.

8. The method according to claim 1, characterized in that Determining the tower group operation rules includes: Obtaining a tower group operation text, and performing relationship extraction on the tower group operation text to obtain an initial rule corresponding to the tower group operation text; Performing priority identification on the initial rule according to the tower group operation text to obtain a target priority corresponding to the initial rule; The initial rule is adjusted according to the target priority to obtain the tower group operation rule.

9. The method according to claim 8, characterized in that The extracting of relations from the tower group operation text to obtain initial rules corresponding to the tower group operation text includes: Perform entity representation on the group tower operation text by using the entity representation layer of the relational extraction model to obtain a first entity representation vector corresponding to the first entity in the group tower operation text and a second entity representation vector corresponding to the second entity; Extracting relationship information between the first entity and the second entity using the entity information extraction layer of the relationship extraction model to obtain a corresponding relationship representation vector between the first entity and the second entity; Using the similarity calculation layer of the relationship extraction model, a similarity calculation is performed based on the first entity representation vector and the relationship representation vector to obtain a first similarity between the first entity representation vector and the relationship representation vector, and a similarity calculation is performed based on the second entity representation vector and the relationship representation vector to obtain a second similarity between the second entity representation vector and the relationship representation vector; Using the context feature extraction layer of the relationship extraction model, extract the context relationship of the first entity to obtain a first context representation vector, and extract the context relationship of the second entity to obtain a second context representation vector; Determine, using the relational feature extraction layer of the relational extraction model, first features corresponding to the first entities according to the first similarity, the second similarity, the first entity representation vector, and the first context representation vector, and determine second features corresponding to the second entities according to the first similarity, the second similarity, the second entity representation vector, and the second context representation vector; Determine a target entity relationship corresponding to the first entity and the second entity according to the first feature and the second feature using the relationship classification layer of the relationship extraction model; Determine the initial rule corresponding to the group tower operation text according to the relationship between the first entity, the second entity and the target entity; The first feature and the second feature are obtained according to the following formula: ; ; ; in, represents the first feature corresponding to the i-th first entity, represents the second feature corresponding to the j-th second entity; represents the first entity representation vector corresponding to the i-th first entity, represents the second entity representation vector corresponding to the j-th second entity, represents an entity relationship vector between the first entity and the second entity, represents a first similarity between the first entity representation vector corresponding to the i-th first entity and the relationship representation vector; represents the second similarity between the second entity representation vector corresponding to the j-th second entity and the relationship representation vector, m represents the first number of associations between the first entity and other entities, represents the tth first context representation vector, n represents the second number of associations between the second entity and other entities, represents the kth second context representation vector, NN represents the neural network corresponding to the relationship feature extraction layer, Indicates that when the first similarity Greater than the second similarity hour, Equal to the first entity representation vector , when the first similarity Less than or equal to the second similarity hour, Equal to the second entity representation vector .

10. A group of towers intelligent collaborative anti-collision system, characterized in that: include: A first acquisition module, used to acquire first image data corresponding to the first tower crane using a first camera and to acquire first sensor data corresponding to the first tower crane using a first sensor; A second acquisition module, used to acquire second image data corresponding to the second tower crane using a second camera and to acquire second sensor data corresponding to the second tower crane using a second sensor; A first processing module, configured to determine first edge information corresponding to the first tower crane according to the first image data and the first sensor data; A second processing module, used for determining second edge information corresponding to the second tower crane according to the second image data and the second sensor data; A position determination module, used to obtain first position information corresponding to the first tower crane using a Beidou positioning system and to obtain second position information corresponding to the second tower crane using the Beidou positioning system; A third processing module, configured to determine first boundary information corresponding to the first tower crane according to the first edge information and the first position information; a fourth processing module, configured to determine second boundary information corresponding to the second tower crane according to the second edge information and the second position information; A state determination module, used to determine a tower group operation rule, and determine a target operation state corresponding to the first tower crane and the second tower crane according to the tower group operation rule combined with the first boundary information and the second boundary information; The data adjustment module is used to perform anti-collision adjustment on the first tower crane and the second tower crane according to the target operating state to obtain target processing results of the first tower crane and the second tower crane.

Images