Intelligent tower crane remote operation method and system based on visual and laser radar fusion recognition

Abstract

The embodiment of the invention provides an intelligent tower crane remote operation method and system based on visual and laser radar fusion recognition, and belongs to the technical field of tower crane operation control. The method comprises the following steps: performing data fusion on first point cloud data, second point cloud data and third point cloud data to obtain target point cloud data; fusing the environment image and the target point cloud data to obtain an initial tower crane three-dimensional model of the target tower crane; obtaining target position information corresponding to a target object in the initial tower crane three-dimensional model according to Beidou positioning, and adding the target position information to the initial tower crane three-dimensional model to obtain a target tower crane three-dimensional model; performing real-time operation state recognition according to the target tower crane three-dimensional model to obtain a target operation state; the target tower crane three-dimensional model and the target operation state are sent to the terminal equipment, so that the terminal equipment displays the target tower crane three-dimensional model and the target operation state to the target user; and a control instruction sent by the terminal equipment is obtained, and the target tower crane is controlled to execute tower crane operation according to the control instruction.

Description

Technical Field

[0001] The present invention relates to the technical field of tower crane operation control, and in particular to an intelligent tower crane remote operation method and system based on vision and laser radar fusion recognition. Background Art

[0002] Tower cranes are indispensable and important equipment in construction sites. They are mainly used for vertical transportation of high-rise buildings. The safety and accuracy of their operation directly affect construction efficiency and personnel safety. However, traditional tower crane operation usually relies on the operator to directly observe and control on the ground or in the operation room. This method not only requires extremely high experience and skills of the operator, but is also restricted by visual limitations, environmental interference, and operating accuracy, which can easily lead to operational errors and thus cause safety accidents.

[0003] In the existing technology, some auxiliary technologies have emerged to improve the safety and efficiency of tower crane operation. However, most of these technologies use vision or laser radar alone, lack effective data fusion methods, and are difficult to fully and accurately reflect the real environment around the tower crane. Especially in complex and changeable construction environments, the data of a single sensor often has limitations and cannot fully cover various factors that may affect operational safety. There is an urgent need for an intelligent tower crane remote operation method that can fuse multi-source sensor data, provide a comprehensive three-dimensional environmental model and real-time status monitoring. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to provide an intelligent tower crane remote operation method and system based on vision and lidar fusion recognition, aiming to solve the problem that most of the related technologies use vision or lidar alone, lack effective data fusion means, and are difficult to fully and accurately reflect the real environment around the tower crane, thereby increasing the difficulty of operation for operators.

[0005] In a first aspect, an embodiment of the present invention provides an intelligent tower crane remote operation method based on vision and laser radar fusion recognition, comprising: Using a first laser radar at a first preset position to collect first point cloud data corresponding to the surrounding environment of the target tower crane, and using a second laser radar at a second preset position to collect second point cloud data corresponding to the surrounding environment of the target tower crane; Using a third laser radar at a third preset position to collect third point cloud data corresponding to the surrounding environment of the target tower crane, and using a visual camera at the third preset position to collect an environmental image corresponding to the surrounding environment of the target tower crane; Performing data fusion on the first point cloud data, the second point cloud data and the third point cloud data to obtain target point cloud data; Fusing the environment image and the target point cloud data to obtain an initial three-dimensional tower crane model corresponding to the surrounding environment of the target tower crane; Obtaining target position information corresponding to the target object in the initial tower crane three-dimensional model according to Beidou positioning, and adding the target position information to the initial tower crane three-dimensional model to obtain a target tower crane three-dimensional model; Performing real-time operation status recognition according to the three-dimensional model of the target tower crane to obtain a target operation status corresponding to the target tower crane; Sending the target tower crane three-dimensional model and the target operating status to a terminal device that is communicatively connected to the target tower crane, so that the terminal device displays the target tower crane three-dimensional model and the target operating status to a target user; The control instruction sent by the terminal device is obtained, and the target tower crane is controlled to perform tower crane operation according to the control instruction.

[0006] In a second aspect, an embodiment of the present invention provides an intelligent tower crane remote operation system based on vision and laser radar fusion recognition, including: A first acquisition module is used to acquire first point cloud data corresponding to the surrounding environment of the target tower crane using a first laser radar at a first preset position, and to acquire second point cloud data corresponding to the surrounding environment of the target tower crane using a second laser radar at a second preset position; A second acquisition module is used to collect third point cloud data corresponding to the surrounding environment of the target tower crane using a third laser radar at a third preset position and to collect an environmental image corresponding to the surrounding environment of the target tower crane using a visual camera at the third preset position; A first fusion module, used for fusing the first point cloud data, the second point cloud data and the third point cloud data to obtain target point cloud data; A second fusion module is used to fuse the environment image and the target point cloud data to obtain an initial tower crane three-dimensional model corresponding to the surrounding environment of the target tower crane; A model adjustment module, used for obtaining target position information corresponding to the target object in the initial tower crane three-dimensional model according to Beidou positioning, and adding the target position information to the initial tower crane three-dimensional model to obtain a target tower crane three-dimensional model; A state recognition module, used for performing real-time operation state recognition according to the three-dimensional model of the target tower crane to obtain a target operation state corresponding to the target tower crane; A data sending module, used for sending the target tower crane three-dimensional model and the target operating status to a terminal device that is communicatively connected to the target tower crane, so that the terminal device displays the target tower crane three-dimensional model and the target operating status to a target user; The instruction execution module is used to obtain the control instruction sent by the terminal device and control the target tower crane to perform tower crane operations according to the control instruction.

[0007] The embodiment of the present invention provides an intelligent tower crane remote operation method and system based on vision and laser radar fusion recognition, the method comprising: using a first laser radar at a first preset position to collect first point cloud data corresponding to the surrounding environment of the target tower crane, and using a second laser radar at a second preset position to collect second point cloud data corresponding to the surrounding environment of the target tower crane; using a third laser radar at a third preset position to collect third point cloud data corresponding to the surrounding environment of the target tower crane and using a visual camera at a third preset position to collect an environmental image corresponding to the surrounding environment of the target tower crane; fusing the first point cloud data, the second point cloud data and the third point cloud data to obtain target point cloud data; fusing the environmental image and the target point cloud data to obtain the initial tower crane 3D model corresponding to the surrounding environment of the target tower crane; obtain the target position information corresponding to the target object in the initial tower crane 3D model according to Beidou positioning, and add the target position information to the initial tower crane 3D model to obtain the target tower crane 3D model; perform real-time operation status recognition according to the target tower crane 3D model to obtain the target operation status corresponding to the target tower crane; send the target tower crane 3D model and the target operation status to the terminal device connected to the target tower crane, so that the terminal device displays the target tower crane 3D model and the target operation status to the target user; obtain the control instructions sent by the terminal device, and control the target tower crane to perform the tower crane operation according to the control instructions. The method collects point cloud data around the target tower crane in all directions and from multiple angles by installing laser radars at different preset positions, including the first point cloud data, the second point cloud data and the third point cloud data. By fusing these point cloud data, more comprehensive and accurate target point cloud data can be generated. Further, combined with the environmental image of the visual camera, a high-precision initial tower crane 3D model can be generated, which significantly improves the accuracy and comprehensiveness of environmental perception. The target position information of the target object is obtained by using Beidou positioning, and it is added to the initial tower crane 3D model to obtain the target tower crane 3D model. This provides the operator with real-time and accurate position information of the target object, greatly improving the accuracy of the operation. The target tower crane 3D model is used to identify the real-time operation status, obtain the target operation status of the target tower crane, and help the operator make more accurate operation decisions. Then, the target tower crane 3D model and the target operation status are sent to the remote terminal device to provide the operator with detailed operation guidance and environmental information, reduce the burden of information processing, and improve operation efficiency. Finally, according to the control instructions sent by the terminal device, the tower crane is automatically controlled to perform the hoisting operation, reduce the operator's work intensity, improve operation efficiency and reduce operation time, thereby significantly improving the overall construction efficiency. It solves the problem that most of the related technologies use vision or laser radar alone, lack effective data fusion methods, and are difficult to fully and accurately reflect the real environment around the tower crane, thereby increasing the difficulty of operator operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] Figure 1 A flowchart of an intelligent tower crane remote operation method based on vision and laser radar fusion recognition provided by an embodiment of the present invention; Figure 2 A schematic diagram of the module structure of an intelligent tower crane remote operation system based on vision and laser radar fusion recognition provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0010] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0011] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0012] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0013] The embodiment of the present invention provides an intelligent tower crane remote operation method and system based on vision and laser radar fusion recognition. The intelligent tower crane remote operation method based on vision and laser radar fusion recognition can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0014] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0015] Please refer to Figure 1 , Figure 1 A flow chart of an intelligent tower crane remote operation method based on vision and laser radar fusion recognition provided in an embodiment of the present invention.

[0016] like Figure 1 As shown, the intelligent tower crane remote operation method based on vision and laser radar fusion recognition includes steps S101 to S108.

[0017] Step S101: using a first laser radar at a first preset position to collect first point cloud data corresponding to the surrounding environment of a target tower crane, and using a second laser radar at a second preset position to collect second point cloud data corresponding to the surrounding environment of the target tower crane.

[0018] Exemplarily, the first preset position and the second preset position are fixedly set at either the boom root or the boom tip of the target tower crane, thereby ensuring that the laser radar can be stably installed and obtain stable environmental data. The boom root is the core supporting part of the tower crane, and the laser radar installed therein can provide comprehensive basic environmental information of the tower crane; while the laser radar at the boom tip can capture environmental data at a longer distance and a higher angle, which is particularly important for environmental perception during high-altitude operations. Therefore, when the first preset position is the boom root of the target tower crane, the second preset position is the boom tip of the target tower crane. When the first preset position is the boom tip of the target tower crane, the second preset position is the boom root of the target tower crane.

[0019] Exemplarily, the first preset position is the boom root of the target tower crane, so that the first laser radar is installed at the first preset position, and the first point cloud data corresponding to the surrounding environment of the target tower crane collected by the first laser radar is obtained. The second preset position is the boom tip of the target tower crane, so that the second laser radar is installed at the second preset position, and the second point cloud data corresponding to the surrounding environment of the target tower crane collected by the second laser radar is obtained.

[0020] Step S102: using a third laser radar at a third preset position to collect third point cloud data corresponding to the surrounding environment of the target tower crane, and using a visual camera at the third preset position to collect an environmental image corresponding to the surrounding environment of the target tower crane.

[0021] For example, the third preset position is flexibly set at the movable position of the boom of the target tower crane, that is, the third preset position essentially refers to the position corresponding to the flexibly movable part in the boom of the tower crane. Such a setting enables the laser radar to adjust its position as the boom moves during the operation of the tower crane, and capture the changes in the environment around the boom in real time. For example, when the boom rotates or extends, the laser radar at the third preset position can adjust its angle and direction at any time to ensure that continuous and accurate environmental data can be provided under different operating conditions.

[0022] Exemplarily, a third laser radar is installed at a third preset position, so as to obtain third point cloud data corresponding to the surrounding environment of the target tower crane collected by the third laser radar.

[0023] For example, through this combination of fixed and flexible settings, the laser radars at the first preset position, the second preset position, and the third preset position can work together to collect environmental data around the tower crane from all directions and angles. The laser radar at the fixed position provides basic and stable perception data, while the laser radar at the movable position is dynamically adjusted according to the operating status of the tower crane, so that the coverage of environmental perception is wider and the data is more comprehensive. This design not only improves the safety of tower crane operation, but also provides operators with more accurate and real-time environmental information, improving overall operating efficiency.

[0024] Exemplarily, the visual camera is arranged at the third preset position at an angle that enables the surrounding environment of the target tower base to be obtained, so that an environmental image corresponding to the surrounding environment of the target tower crane is collected by the visual camera.

[0025] Step S103: fusing the first point cloud data, the second point cloud data and the third point cloud data to obtain target point cloud data.

[0026] Exemplarily, the collected first point cloud data, second point cloud data, and third point cloud data are denoised to remove invalid points and noise points caused by environmental interference, equipment noise, etc., to ensure the purity of the data.

[0027] Exemplarily, the denoised first point cloud data, the second point cloud data and the third point cloud data are initially registered, and the point cloud data collected at different positions are aligned to the same coordinate system for subsequent precise registration and fusion. Thus, a precise registration algorithm, such as the ICP (iterative closest point) algorithm or the NDT (normal distribution transformation) algorithm, is used to accurately register the first point cloud data, the second point cloud data and the third point cloud data to ensure the precise positioning and alignment of the point cloud data in space. Thus, the aligned first point cloud data, the second point cloud data and the third point cloud data are reorganized and optimized to generate target point cloud data. The target point cloud data should contain comprehensive information from the first point cloud data, the second point cloud data and the third point cloud data, covering the comprehensive area of ​​the surrounding environment of the target tower crane.

[0028] In some embodiments, the data fusion of the first point cloud data, the second point cloud data, and the third point cloud data to obtain target point cloud data includes: performing data clustering on the first point cloud data to obtain a first clustering result and a first target type corresponding to the first clustering result; performing data clustering on the second point cloud data to obtain a second clustering result and a second target type corresponding to the second clustering result; performing data clustering on the third point cloud data to obtain a third clustering result and a third target type corresponding to the third clustering result; performing target recognition on the environment image using a target recognition model to obtain a corresponding target recognition type; and performing data clustering on the environment image using the first target type according to the target recognition type. The first clustering result is screened to obtain a first point cloud cluster; the second clustering result is screened using the second target type according to the target recognition type to obtain a second point cloud cluster; the third clustering result is screened using the third target type according to the target recognition type to obtain a third point cloud cluster; the first point cloud cluster and the third point cloud cluster are data aligned to obtain a first alignment result, and the second point cloud cluster and the third point cloud cluster are data aligned to obtain a second alignment result; the first point cloud data, the second point cloud data and the third point cloud data are data fused according to the first alignment result and the second alignment result to obtain the target point cloud data.

[0029] Exemplarily, a clustering algorithm, such as K-means, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) or Mean Shift, is selected to cluster the first point cloud data to obtain a first clustering result, thereby classifying the first clustering result and determining a first target type corresponding to each clustering cluster.

[0030] Exemplarily, the above clustering steps are repeated for the second point cloud data and the third point cloud data, respectively, to obtain a second clustering result and a third clustering result, and determine the corresponding second target type and third target type.

[0031] For example, a target recognition model such as a deep learning model (such as YOLO, Faster R-CNN) is used to perform target recognition on the environment image to obtain the corresponding target recognition type. The target recognition type should match the target type in the clustering result of the point cloud data.

[0032] Exemplarily, according to the target recognition type, the first target type matching the target recognition type in the first clustering result is screened to obtain the first point cloud cluster. According to the target recognition type, the second target type matching the target recognition type in the second clustering result is screened to obtain the second point cloud cluster. According to the target recognition type, the third target type matching the target recognition type in the third clustering result is screened to obtain the third point cloud cluster.

[0033] Exemplarily, a point cloud registration algorithm, such as ICP or NDT, is used to align the first point cloud cluster and the third point cloud cluster to obtain a first alignment result. Ensure that the two first point cloud clusters and the third point cloud cluster are aligned in the same coordinate system. Use the same registration algorithm to align the second point cloud cluster and the third point cloud cluster to obtain a second alignment result. Ensure that the second point cloud cluster and the third point cloud cluster are aligned in the same coordinate system. Among them, the coordinate system corresponding to the third point cloud cluster is used as the target coordinate system in both the first alignment result and the second alignment result.

[0034] Exemplarily, the first point cloud data, the second point cloud data and the third point cloud data are fused based on the feature point matching algorithm in combination with the first alignment result to obtain the target point cloud data. The target point cloud data should contain comprehensive information from the first point cloud data, the second point cloud data and the third point cloud data, covering the comprehensive area of ​​the surrounding environment of the target tower crane.

[0035] In some embodiments, the use of a target recognition model to perform target recognition on the environmental image to obtain a corresponding target recognition type includes: using the input layer of the target recognition model to perform a preprocessing operation on the environmental image to obtain a preprocessed image; using the feature extraction layer of the target recognition model to perform feature extraction on the preprocessed image to obtain initial feature information; using the variable convolutional layer of the target recognition model to add a learnable offset and an adjustment factor at the sampling position to adaptively perform feature sampling on the initial feature information to obtain target feature information; using the target recognition layer of the target recognition model to perform target recognition on the target feature information to obtain the target recognition type.

[0036] Exemplarily, the environment image is input to the input layer of the target recognition model. The environment image is preprocessed to improve image quality and model recognition performance. The preprocessing operation includes but is not limited to image scaling, normalization and other operations.

[0037] Exemplarily, the feature extraction layer of the target recognition model is used to extract features from the preprocessed image. The feature extraction layer usually includes multiple layers of convolution operations to extract low-level and high-level features in the image. The features extracted by the convolution operation are called initial feature information. These features usually include the basic structure and local detail information of the image.

[0038] Exemplarily, a variable convolution layer of the target recognition model is used to add a learnable offset and adjustment factor to the sampling position. The variable convolution layer adaptively adjusts the position and shape of the convolution kernel by learning the offset and adjustment factor to adapt to different image features and structures, thereby sampling the initial feature information according to the offset and adjustment factor to obtain the target feature information. The target feature information is a feature that has been adaptively adjusted and can better capture the key information and structure in the image.

[0039] Exemplarily, the target recognition layer of the target recognition model is used to perform target recognition on the target feature information. The target recognition layer usually includes a fully connected layer and a classifier, which is used to convert the feature information into a target recognition type. Thus, the target feature information is classified through the target recognition layer to obtain the final target recognition type. The target recognition type should match the pre-defined target category.

[0040] Specifically, the input layer of the target recognition model is used to preprocess the environmental image, and the feature extraction layer, variable convolution layer and target recognition layer are used to extract features and recognize targets in the image, so as to obtain accurate target recognition types and provide reliable environmental perception information for subsequent point cloud data processing and tower crane operation.

[0041] Step S104: fusing the environment image and the target point cloud data to obtain an initial three-dimensional tower crane model corresponding to the surrounding environment of the target tower crane.

[0042] Exemplarily, image target detection is performed on the environment image to obtain a first target position corresponding to the environment image and a first target type corresponding to the first target position. Three-dimensional target detection is performed on the target point cloud data to obtain a second target position corresponding to the target point cloud data and a second target type corresponding to the second target position.

[0043] Exemplarily, target coordinate point detection is performed on the second target position to obtain a minimum rectangular frame that can cover the target object corresponding to the second target position and obtain a third target position corresponding to the minimum rectangular frame. In addition, if the visual camera and the third preset position are in the same position, it can be determined that the third target position and the first target position are in the same coordinate system.

[0044] Exemplarily, the degree of overlap between the first target position and the third target position is calculated, and then when the degree of overlap is greater than a preset value, the first target type and the second target type are compared to see whether they are the same. Thus, when the degree of overlap between the first target position and the third target position is greater than the preset value and the first target type and the second target type are the same, the target type corresponding to the associated point cloud data corresponding to the second target position is determined; when the degree of overlap between the first target position and the third target position is less than or equal to the preset value or the first target type and the second target type are different, the target type corresponding to the associated point cloud data corresponding to the second target position cannot be determined, and then target detection is re-performed on the target point cloud data and target detection is re-performed on the environmental image until the target type corresponding to the associated point cloud data corresponding to the second target position is determined.

[0045] Exemplarily, an initial tower crane three-dimensional model corresponding to the surrounding environment of the target tower crane is determined according to the target type and the target point cloud data.

[0046] In some embodiments, the fusing of the environmental image and the target point cloud data to obtain an initial three-dimensional tower crane model corresponding to the surrounding environment of the target tower crane includes: performing image enhancement on the environmental image to obtain a corresponding target image; performing edge recognition on the target image to obtain target edge information corresponding to the target image, and performing target recognition on the target edge information according to a convolutional neural network to obtain a first recognition result corresponding to the target image; performing target recognition on the target point cloud data to obtain a second recognition result corresponding to the target point cloud data; and performing data fusion on the environmental image and the target point cloud data according to the first recognition result and the second recognition result to obtain the initial three-dimensional tower crane model.

[0047] Exemplarily, an image enhancement operation is performed on the environment image to improve the image quality and the accuracy of target recognition. Common image enhancement methods include but are not limited to contrast enhancement, brightness adjustment, denoising, and sharpening. Thus, a target image is obtained.

[0048] Exemplarily, the Canny edge detection, Sobel operator, Laplacian operator, etc. are used to perform edge recognition on the target image to extract target edge information in the target image.

[0049] Exemplarily, a convolutional neural network (CNN) is used to identify the target edge information. CNN extracts high-level features in the image through multi-layer convolution and pooling operations, and obtains the first recognition result corresponding to the environment image by classifying the target through a fully connected layer and a classifier. The first recognition result includes the category and location information of the target in the environment image. That is, the first recognition result includes the first target location and the first target type corresponding to the first target location.

[0050] Exemplarily, a target recognition algorithm such as a deep learning model (such as PointNet, PointNet++) is used to perform target recognition on the target point cloud data to obtain a second recognition result, and the second recognition result includes the category and location information of the target in the target point cloud data. That is, the second recognition result includes the second target location and the second target type corresponding to the second target location.

[0051] Exemplarily, target coordinate point detection is performed on the second target position to obtain a minimum rectangular frame that can cover the target object corresponding to the second target position and obtain a third target position corresponding to the minimum rectangular frame. In addition, if the visual camera and the third preset position are in the same position, it can be determined that the third target position and the first target position are in the same coordinate system.

[0052] Exemplarily, the degree of overlap between the first target position and the third target position is calculated, and then when the degree of overlap is greater than a preset value, the first target type and the second target type are compared to see whether they are the same. Thus, when the degree of overlap between the first target position and the third target position is greater than the preset value and the first target type and the second target type are the same, the target type corresponding to the associated point cloud data corresponding to the second target position is determined; when the degree of overlap between the first target position and the third target position is less than or equal to the preset value or the first target type and the second target type are different, the target type corresponding to the associated point cloud data corresponding to the second target position cannot be determined, and then target detection is re-performed on the target point cloud data and target detection is re-performed on the environmental image until the target type corresponding to the associated point cloud data corresponding to the second target position is determined.

[0053] Exemplarily, an initial tower crane three-dimensional model corresponding to the surrounding environment of the target tower crane is determined according to the target type and the target point cloud data.

[0054] In some embodiments, the image enhancement of the environmental image to obtain the corresponding target image includes: extracting brightness features of the environmental image to obtain a corresponding first brightness feature and extracting saturation features of the environmental image to obtain a corresponding first saturation feature; performing data enhancement on the first brightness feature to obtain a second brightness feature, and correcting the second brightness feature using a gamma correction algorithm to obtain a third brightness feature; determining a target window, and obtaining a window brightness feature corresponding to the third brightness feature under the target window; performing regional histogram equalization on the window brightness feature to obtain a corresponding target brightness feature, and recombining according to the target brightness feature to obtain a fourth brightness feature; performing bilateral filtering on the first saturation feature to obtain a second saturation feature, and performing a stretching transformation on the second saturation feature to obtain a third saturation feature; and obtaining the corresponding target image according to the fourth brightness feature and the third saturation feature.

[0055] Exemplarily, the environment image is converted into the HSV space to perform brightness feature extraction to obtain a first brightness feature corresponding to the environment image, and saturation feature extraction is performed to obtain a first saturation feature corresponding to the environment image.

[0056] Exemplarily, a data enhancement operation is performed on the first brightness feature, such as contrast enhancement, brightness adjustment, etc., to obtain the second brightness feature. Data enhancement can improve the contrast and detail information of the environment image.

[0057] Exemplarily, a gamma correction algorithm is applied to the second brightness feature to perform correction processing to obtain a third brightness feature. Gamma correction can adjust the brightness and contrast of an image to improve the visual effect of the image.

[0058] Exemplarily, a target window is determined, thereby extracting a window brightness feature of the third brightness feature under the target window. The window brightness feature only contains brightness information within the target window. Thus, the window brightness feature is subjected to regional histogram equalization processing to make the brightness feature within the target window more balanced, thereby obtaining a target brightness feature. Regional histogram equalization can enhance the contrast and details of the target area. Finally, the third brightness feature is recombined according to the target brightness feature to obtain a fourth brightness feature.

[0059] Exemplarily, the first saturation feature is subjected to bilateral filtering to obtain a second saturation feature. Bilateral filtering can smooth the image while retaining edge information, thereby improving the smoothness and detail quality of the image. The second saturation feature is then subjected to stretching transformation to obtain a third saturation feature. The stretching transformation can adjust the saturation range of the image, thereby improving the color saturation and visual effect of the image.

[0060] Exemplarily, the ambient image is converted into the HSV space to perform hue feature extraction, and the hue feature corresponding to the ambient image is obtained, and then feature fusion is performed according to the fourth brightness feature, the third saturation feature and the hue feature to obtain a new HSV image, and then the new HSV image is converted into the RGB space to obtain the target image.

[0061] In some embodiments, the edge recognition of the target image to obtain target edge information corresponding to the target image includes: performing a wavelet transform on the target image to obtain a first transform result of the target image in the horizontal and vertical directions; performing a wavelet transform on the target image to obtain a second transform result of the target image in the horizontal direction and performing a wavelet transform on the target image to obtain a third transform result of the target image in the vertical direction; determining a target gradient corresponding to the target image based on the first transform result, the second transform result, and the third transform result; determining a segmentation threshold corresponding to the target image based on the target gradient, and determining a target edge point corresponding to the target image based on the segmentation threshold; and performing data clustering on the target edge points to obtain target edge information corresponding to the target image.

[0062] Exemplarily, a wavelet transform is performed on a target image to obtain a first transform result of the target image in the horizontal direction and the vertical direction. The first transform result contains low-frequency information of the target image in both the horizontal direction and the vertical direction. A wavelet transform is performed on the target image to obtain a second transform result of the target image in the horizontal direction. The second transform result mainly contains high-frequency information of the target image in the horizontal direction. A wavelet transform is performed on the target image to obtain a third transform result of the target image in the vertical direction. The third transform result mainly contains high-frequency information of the target image in the vertical direction.

[0063] Exemplarily, a first optical component of the target image in the horizontal direction and a second optical component in the vertical direction are obtained, and then the first optical component is squared and the second optical component is squared, and then the square root is taken to obtain the target optical component. The second transformation result and the third transformation result are summed, and then the summation result is multiplied by the first transformation result to obtain the target transformation result, and then the target optical component is divided by the target transformation result to obtain the target gradient of the target image at the corresponding position.

[0064] Exemplarily, a gradient calculation is performed on the target image to obtain the gradient information of each pixel in the target image. The gradient information of each pixel is calculated by ratio with the gradient information of its adjacent pixels. The purpose of the ratio calculation is to highlight the area with significant gradient changes in the image, so as to better identify the edge points. After the ratio calculation, a non-maximum suppression operation is further performed. The purpose of the non-maximum suppression operation is to retain only the pixels with the largest local gradient, thereby reducing the number of edge points and making the edges clearer and more accurate. After the non-maximum suppression operation, the pixels with the largest local gradient retained are called maximum points. These maximum points may be potential edge points, but further processing is required to determine the true edge. The arg function is calculated on the maximum points to obtain the segmentation threshold. The arg function is usually used to calculate the relative position of the extreme point in the gradient direction, so as to determine the appropriate segmentation threshold. The segmentation threshold is used to distinguish edge points from non-edge points.

[0065] Exemplarily, the target image is edge identified according to the segmentation threshold, and when the pixel value corresponding to the target image is greater than or equal to the segmentation threshold, the pixel value is determined as the target edge point corresponding to the target image. The target edge point accurately depicts the outline and boundary of the target object.

[0066] Exemplarily, the extracted target edge points are clustered using K-means clustering, DBSCAN clustering, hierarchical clustering, etc. By clustering the target edge points, the target edge information corresponding to the target image is obtained. The target edge information includes the contour, structure and shape information of the target.

[0067] In some embodiments, the data clustering of the target edge points to obtain the target edge information corresponding to the target image includes: determining an initial cluster center and obtaining distance information between the target edge point and the initial cluster center; determining the target probability that the target edge point belongs to the initial cluster center according to the distance information; calculating the first membership corresponding to the target edge point belonging to the initial cluster center, and determining the clustering target value corresponding to the initial cluster center according to the distance information and the first membership; updating the first membership according to the clustering target value and the target probability to obtain a second membership; updating the initial cluster center according to the second membership to obtain a target cluster center; performing data clustering on the target edge points according to the target cluster center to obtain the target edge information corresponding to the target image; wherein the clustering target value and the second membership are obtained according to the following formula:

[0068]

[0069] in, represents the clustering target value, cer represents the number of the initial clustering centers, m represents the number of the target edge points, represents the segmentation threshold, represents the pixel value corresponding to the jth target edge point, represents the number of pixel values ​​corresponding to the jth target edge point, Indicates that the j-th target edge point belongs to the first membership degree of the i-th initial cluster center, represents the pixel value corresponding to the i-th initial cluster center; The second membership degree indicates that the j-th target edge point belongs to the i-th initial cluster center, represents the probability that the j-th target edge point belongs to the target corresponding to the i-th initial cluster center, Indicates the minimum value of the target probability that the j-th target edge point belongs to the target corresponding to the i-th initial cluster center.

[0070] Exemplarily, the initial cluster centers are determined in the target edge points. The initial cluster centers can be determined by random selection, K-means++ initialization or other initialization methods. The number of initial cluster centers is preset according to actual needs.

[0071] Exemplarily, the distance information between each target edge point and each initial cluster center is calculated using Euclidean distance, Manhattan distance or other distance measurement methods to quantify the similarity between the target edge point and the initial cluster center.

[0072] Exemplarily, the calculated distance information is used to determine the target probability that each target edge point belongs to each initial cluster center. The target probability can be calculated by a Gaussian distribution function or other probability distribution functions, so that target edge points with closer distances have a higher probability of belonging to the corresponding cluster center.

[0073] Exemplarily, the first membership of each target edge point to each initial cluster center is calculated. The first membership can be calculated by target probability or other membership functions, indicating the degree of belonging of the target edge point to each cluster center.

[0074] Exemplarily, the clustering target value corresponding to each initial clustering center is calculated according to the distance information and the first membership degree. The clustering target value can reflect the clustering effect of the initial clustering center. The clustering target value is obtained according to the following formula:

[0075] in, represents the clustering target value, cer represents the number of initial cluster centers, and m represents the number of target edge points. represents the segmentation threshold, represents the pixel value corresponding to the jth target edge point, Represents the number of pixel values ​​corresponding to the jth target edge point, Indicates the first membership degree of the jth target edge point to the ith initial cluster center, Represents the pixel value corresponding to the i-th initial cluster center.

[0076] Exemplarily, by calculating the clustering target value, the clustering effect of the initial clustering center can be quantified. The clustering target value reflects the performance of each initial clustering center in the clustering process and provides an objective indicator for evaluating the clustering effect. The calculation of the clustering target value is based on distance information and the first membership, which helps to optimize the position of the initial clustering center. By adjusting the clustering target value, the clustering center can be closer to the actual clustering center position, thereby improving the accuracy of clustering. The calculation formula of the clustering target value comprehensively considers the pixel value of the target edge point, the segmentation threshold, the first membership and the pixel value of the initial clustering center. These factors work together to enable the clustering target value to more accurately reflect the clustering effect, thereby improving the accuracy of clustering. Through the calculation of the clustering target value, the stability of the clustering process can be enhanced. As a dynamically adjusted parameter, the clustering target value can provide feedback in each iteration to ensure that the clustering process is carried out in a more optimal direction and reduce the instability of the clustering results.

[0077] Exemplarily, the first membership is updated according to the clustering target value and the target probability to obtain the second membership. The second membership is obtained according to the following formula:

[0078] in, represents the clustering target value, cer represents the number of initial cluster centers, and m represents the number of target edge points. Indicates the second membership degree of the jth target edge point to the ith initial cluster center, It indicates the probability that the jth target edge point belongs to the target corresponding to the i-th initial cluster center. Indicates the minimum value of the target probability that the j-th target edge point belongs to the i-th initial cluster center.

[0079] Exemplarily, the first membership is updated by the clustering target value and the target probability, and the obtained second membership can more accurately reflect the degree of belonging of the target edge point to the cluster center. Compared with the first membership, the second membership integrates more information, making the calculation of the membership more accurate. The calculation of the second membership takes into account the clustering target value and the target probability, which helps to optimize the clustering results. By adjusting the membership, the cluster center is closer to the actual cluster center position, thereby improving the accuracy and stability of clustering. The updating process of the second membership dynamically adjusts the degree of belonging of the target edge point to the cluster center, making the clustering process more stable. By updating the membership, feedback can be provided in each iteration to ensure that the clustering process is moving in a more optimal direction.

[0080] Exemplarily, according to the second membership degree, the initial cluster center is updated to obtain the target cluster center. The updating method is to perform weighted average according to the contribution of the target edge point to each cluster center, so that the cluster center is closer to the actual cluster center position. The target edge point is clustered using the updated target cluster center until the target cluster center no longer changes. Thus, the target edge information corresponding to the target image is obtained by clustering the target edge point according to the target cluster center.

[0081] Step S105: obtaining target position information corresponding to the target object in the initial tower crane three-dimensional model according to Beidou positioning, and adding the target position information to the initial tower crane three-dimensional model to obtain a target tower crane three-dimensional model.

[0082] Exemplarily, the target position information of the target object (such as a hook, a boom, etc.) in the initial tower crane 3D model is obtained according to Beidou positioning, and then the relative position relationship between the remaining objects in the initial tower crane 3D model and the target object is obtained, so that the associated position relationship corresponding to the remaining objects in the initial tower crane 3D model is determined according to the relative position relationship and the target position information, and then the associated position relationship and the target position information are added to the initial tower crane 3D model to obtain the target tower crane 3D model. That is, the target tower crane 3D model contains the real position information corresponding to each target object.

[0083] In some embodiments, obtaining the target position information corresponding to the target object in the initial tower crane three-dimensional model based on Beidou positioning includes: obtaining the target hanging object corresponding to the target tower crane from the initial tower crane three-dimensional model; obtaining the hanging object position information corresponding to the target hanging object using the Beidou positioning; and calculating the position of the target object in the initial tower crane three-dimensional model based on the hanging object position information to obtain the target position information corresponding to the target object.

[0084] Exemplarily, a target hoisted object corresponding to the target tower crane is identified from the initial tower crane 3D model. The target hoisted object generally refers to an object being hoisted by the tower crane, such as building materials, mechanical equipment, etc. The target hoisted object is determined by an identifier or feature point in the initial tower crane 3D model.

[0085] Exemplarily, the Beidou positioning system is started to ensure its normal operation. The Beidou positioning system determines the object position information of the target object by receiving satellite signals. The object position information generally includes longitude, latitude and altitude to form a three-dimensional coordinate point.

[0086] Exemplarily, the relative position between the target object and the target hanging object in the initial tower crane three-dimensional model is calculated to ensure that the position of the target object is consistent with the position of the hanging object, and then the target position information of the target object in the Beidou positioning system is determined based on the relative position and the hanging object position information.

[0087] Specifically, the target hoist corresponding to the target tower crane is obtained from the initial tower crane 3D model, the target hoist position information of the target hoist is obtained using the Beidou positioning system, and the position of the target object in the initial tower crane 3D model is calculated based on the hoist position information, and finally the target position information corresponding to the target object is obtained. This process improves the accuracy and practicality of the initial tower crane 3D model and provides reliable data support for the operation and monitoring of the tower crane.

[0088] Step S106: performing real-time operation status recognition according to the three-dimensional model of the target tower crane to obtain a target operation status corresponding to the target tower crane.

[0089] Exemplarily, the target tower crane 3D model that has been acquired and contains the target position information is loaded. The target tower crane 3D model is an important carrier of the tower crane digital twin, which can accurately reflect the geometric structure of the tower crane and the spatial relationship between the components. The target tower crane 3D model should have accurately embedded the real-time position information of the target object (such as hook, boom, etc.), laying the foundation for the subsequent operation status identification and analysis.

[0090] Exemplarily, key operating state parameters are extracted from the three-dimensional model of the target tower crane. These parameters are quantitative indicators describing the operating state of the tower crane at a specific moment, mainly including: the angle of the boom, the height of the hook, the position of the hanging object, the speed of the tower crane, and the spatial relationship between the hanging object and the surrounding objects, so that after obtaining the operating state parameters at each moment, the change rate of these parameters at adjacent moments is calculated. The change rate is an indicator that describes how fast the operating state of the tower crane changes over time, mainly including: the angle change rate of the boom, the height change rate of the hook, the moving speed of the hanging object, and the degree of change corresponding to the spatial relationship between the hanging object and the surrounding objects, and then the calculated operating state parameter change rate is input into the pre-trained state prediction model. The model uses a machine learning algorithm or a deep learning network, and can accurately predict the operating state of the tower crane at the next moment by learning a large amount of historical data. The change rate data input by the model include the boom angle change rate, the hook height change rate, the hanging object moving speed, etc., and the output is the target operating state of the tower crane, such as the target operating state including but not limited to safe operation, abnormality, etc.

[0091] In some embodiments, the target tower crane three-dimensional model includes a first tower crane three-dimensional model corresponding to a first moment and a second tower crane three-dimensional model corresponding to a second moment adjacent to the first moment, and the real-time operation status identification based on the target tower crane three-dimensional model obtains the target operation status corresponding to the target tower crane, including: obtaining first feature information corresponding to the target hanging object from the first tower crane three-dimensional model and obtaining second feature information corresponding to the target hanging object from the second tower crane three-dimensional model; determining the change angle corresponding to each associated position in the target hanging object according to the first feature information and the second feature information; and determining the maximum movement angle corresponding to the target hanging object according to the change angle; performing difference calculation based on the change angle and the maximum movement angle to determine the angle difference corresponding to the associated position in the target hanging object; determining the change weight corresponding to the associated position according to the change angle and the angle difference, and determining the target change value corresponding to the target tower crane according to the change weight, the first feature information and the second feature information; determining the target operation status corresponding to the target tower crane according to the target change value.

[0092] Exemplarily, the target tower crane three-dimensional model includes a first tower crane three-dimensional model corresponding to a first moment and a second tower crane three-dimensional model corresponding to a second moment adjacent to the first moment.

[0093] Exemplarily, a first tower crane three-dimensional model is loaded, and first characteristic information of the target hanging object in the model is extracted. The first characteristic information may include the position, posture, shape, and relative position of the target hanging object with respect to other parts of the tower crane.

[0094] Exemplarily, the second tower crane three-dimensional model is loaded, and the second characteristic information of the target hanging object in the model is also extracted, ensuring that the extracted second characteristic information is consistent with the information extracted in the first tower crane three-dimensional model in terms of definition and representation.

[0095] Exemplarily, the first characteristic information and the second characteristic information of the target hanging object at each associated position in the first tower crane three-dimensional model and the second tower crane three-dimensional model are compared. The associated position may be a key point on the target hanging object or a contact point with other parts of the tower crane, etc. By calculating the angle change of the associated position in the two models, the change angle of the target hanging object at each associated position is determined.

[0096] For example, according to the change angle of each associated position calculated in the previous step, the maximum movement angle of the target hanging object in all associated positions is determined. This angle reflects the maximum rotation or tilt of the hanging object during the entire movement process.

[0097] Exemplarily, the difference between the change angle of each associated position and the maximum movement angle of the target hanging object is calculated to obtain the angle difference of each associated position. The angle difference reflects the size difference of each associated position relative to the maximum movement angle.

[0098] Exemplarily, the change weight of each associated position is determined according to the calculated angle difference. The change weight can be represented by the absolute value of the angle difference or other related metrics to measure the importance and contribution of each associated position change.

[0099] Exemplarily, the target change value of the target tower crane is calculated by comprehensively considering the change weight of each associated position, the first feature information and the second feature information. The target change value is a comprehensive evaluation of the overall change of the target tower crane, which can reflect the dynamic characteristics and state changes of the tower crane during movement.

[0100] For example, the change information between the second feature information and the first feature information is calculated, and then a weighted sum is performed according to the change weight and the change information to obtain the target change value of the target tower crane.

[0101] Exemplarily, the calculated target change value is compared with a preset threshold. When the target change value is greater than or equal to the preset threshold, the target operating state is determined to be abnormal operation; when the target change value is less than the preset threshold, the target operating state is determined to be normal operation.

[0102] Specifically, characteristic information of the target hoisted object is obtained from the first tower crane 3D model and the second tower crane 3D model, and the target operation state of the target tower crane is determined. This process improves the real-time and accuracy of tower crane operation, and provides reliable support for the safe and efficient operation of tower cranes.

[0103] Step S107: sending the target tower crane three-dimensional model and the target operating status to a terminal device that is communicatively connected to the target tower crane, so that the terminal device displays the target tower crane three-dimensional model and the target operating status to a target user.

[0104] Exemplarily, the three-dimensional model of the target tower crane and the target operating status are sent to a terminal device that is communicatively connected to the target tower crane, so that the terminal device displays the three-dimensional model of the target tower crane and the target operating status to the target user, and then the target user can obtain the three-dimensional model of the target tower crane and the target operating status according to the terminal device, and then operate the tower crane according to the three-dimensional model of the target tower crane and the target operating status, and determine the corresponding control instructions.

[0105] Step S108: obtaining the control instruction sent by the terminal device, and controlling the target tower crane to perform tower crane operation according to the control instruction.

[0106] Exemplarily, the control instructions sent by the terminal device are obtained, and then the target tower crane is controlled to perform the tower crane operation according to the control instructions.

[0107] See also Figure 2 , Figure 2An intelligent tower crane remote operation system 200 based on vision and laser radar fusion recognition is provided in the embodiment of the present application. The intelligent tower crane remote operation system 200 based on vision and laser radar fusion recognition includes a first acquisition module 201, a second acquisition module 202, a first fusion module 203, a second fusion module 204, a model adjustment module 205, a state recognition module 206, a data sending module 207, and an instruction execution module 208. The first acquisition module 201 is used to use a first laser radar at a first preset position to collect first point cloud data corresponding to the surrounding environment of the target tower crane, and use a second laser radar at a second preset position to collect second point cloud data corresponding to the surrounding environment of the target tower crane; the second acquisition module 202 is used to use a third laser radar at a third preset position to collect third point cloud data corresponding to the surrounding environment of the target tower crane and use a visual camera at the third preset position to collect an environmental image corresponding to the surrounding environment of the target tower crane; the first fusion module 203 is used to collect the first point cloud data, the second point cloud data, and the second point cloud data. The cloud data and the third point cloud data are fused to obtain the target point cloud data; the second fusion module 204 is used to fuse the environmental image and the target point cloud data to obtain the initial tower crane three-dimensional model corresponding to the surrounding environment of the target tower crane; the model adjustment module 205 is used to obtain the target position information corresponding to the target object in the initial tower crane three-dimensional model according to Beidou positioning, and add the target position information to the initial tower crane three-dimensional model to obtain the target tower crane three-dimensional model; the state recognition module 206 is used to perform real-time operation state recognition according to the target tower crane three-dimensional model to obtain the target operation state corresponding to the target tower crane; the data sending module 207 is used to send the target tower crane three-dimensional model and the target operation state to the terminal device connected to the target tower crane, so that the terminal device displays the target tower crane three-dimensional model and the target operation state to the target user; the instruction execution module 208 is used to obtain the control instruction sent by the terminal device, and control the target tower crane to perform tower crane operation according to the control instruction.

[0108] In some implementations, the first fusion module 203, during the process of fusing the first point cloud data, the second point cloud data, and the third point cloud data to obtain target point cloud data, executes: Performing data clustering on the first point cloud data to obtain a first clustering result and a first target type corresponding to the first clustering result; Performing data clustering on the second point cloud data to obtain a second clustering result and a second target type corresponding to the second clustering result; Performing data clustering on the third point cloud data to obtain a third clustering result and a third target type corresponding to the third clustering result; Using a target recognition model to perform target recognition on the environment image to obtain a corresponding target recognition type; Using the first target type to filter the first clustering result according to the target recognition type to obtain a first point cloud cluster; Using the second target type to filter the second clustering result according to the target recognition type to obtain a second point cloud cluster; Using the third target type to filter the third clustering result according to the target recognition type to obtain a third point cloud cluster; Performing data alignment on the first point cloud cluster and the third point cloud cluster to obtain a first alignment result, and performing data alignment on the second point cloud cluster and the third point cloud cluster to obtain a second alignment result; The target point cloud data is obtained by performing data fusion on the first point cloud data, the second point cloud data and the third point cloud data according to the first alignment result and the second alignment result.

[0109] In some implementations, the first fusion module 203 performs, during the process of using the target recognition model to perform target recognition on the environment image to obtain the corresponding target recognition type: Using the input layer of the target recognition model to perform a preprocessing operation on the environment image to obtain a preprocessed image; Using the feature extraction layer of the target recognition model to extract features from the preprocessed image to obtain initial feature information; Using the variable convolutional layer of the target recognition model to add a learnable offset and an adjustment factor at the sampling position to adaptively perform feature sampling on the initial feature information to obtain target feature information; The target recognition layer of the target recognition model is used to perform target recognition on the target feature information to obtain the target recognition type.

[0110] In some implementations, the second fusion module 204 performs, during the process of fusing the environment image and the target point cloud data to obtain the initial tower crane three-dimensional model corresponding to the surrounding environment of the target tower crane: Performing image enhancement on the environment image to obtain a corresponding target image; Performing edge recognition on the target image to obtain target edge information corresponding to the target image, and performing target recognition on the target edge information according to a convolutional neural network to obtain a first recognition result corresponding to the target image; Performing target recognition on the target point cloud data to obtain a second recognition result corresponding to the target point cloud data; The initial tower crane three-dimensional model is obtained by performing data fusion on the environment image and the target point cloud data according to the first recognition result and the second recognition result.

[0111] In some implementations, the second fusion module 204 performs, during the process of performing image enhancement on the environment image to obtain the corresponding target image: Extracting brightness features from the environment image to obtain a corresponding first brightness feature and extracting saturation features from the environment image to obtain a corresponding first saturation feature; Performing data enhancement on the first brightness feature to obtain a second brightness feature, and performing correction processing on the second brightness feature using a gamma correction algorithm to obtain a third brightness feature; Determine a target window, and obtain a window brightness feature corresponding to the third brightness feature under the target window; Performing regional histogram equalization processing on the window brightness feature to obtain a corresponding target brightness feature, and recombining according to the target brightness feature to obtain a fourth brightness feature; Performing bilateral filtering on the first saturation feature to obtain a second saturation feature, and performing stretching transformation on the second saturation feature to obtain a third saturation feature; The corresponding target image is obtained according to the fourth brightness feature and the third saturation feature.

[0112] In some implementations, the second fusion module 204 performs, during the process of performing edge recognition on the target image to obtain target edge information corresponding to the target image: Performing wavelet transform on the target image to obtain first transform results of the target image in the horizontal direction and the vertical direction; Performing a wavelet transform on the target image to obtain a second transform result of the target image in the horizontal direction and performing a wavelet transform on the target image to obtain a third transform result of the target image in the vertical direction; Determine a target gradient corresponding to the target image according to the first transformation result, the second transformation result, and the third transformation result; Determine a segmentation threshold corresponding to the target image according to the target gradient, and determine a target edge point corresponding to the target image according to the segmentation threshold; Data clustering is performed on the target edge points to obtain target edge information corresponding to the target image.

[0113] In some implementations, the second fusion module 204 performs, during the process of performing data clustering on the target edge points to obtain the target edge information corresponding to the target image: Determine an initial cluster center, and obtain distance information between the target edge point and the initial cluster center; Determine, according to the distance information, the probability that the target edge point belongs to the target corresponding to the initial cluster center; Calculating a first degree of membership of the target edge point to the initial cluster center, and determining a cluster target value corresponding to the initial cluster center according to the distance information and the first degree of membership; Update the first membership degree according to the clustering target value and the target probability to obtain a second membership degree; updating the initial cluster center according to the second membership degree to obtain a target cluster center; Performing data clustering on the target edge points according to the target cluster center to obtain the target edge information corresponding to the target image; The clustering target value and the second degree of membership are obtained according to the following formula:

[0114]

[0115] in, represents the clustering target value, cer represents the number of the initial clustering centers, m represents the number of the target edge points, represents the segmentation threshold, represents the pixel value corresponding to the jth target edge point, represents the number of pixel values ​​corresponding to the jth target edge point, Indicates that the j-th target edge point belongs to the first membership degree of the i-th initial cluster center, represents the pixel value corresponding to the i-th initial cluster center; The second membership degree indicates that the j-th target edge point belongs to the i-th initial cluster center, represents the probability that the j-th target edge point belongs to the target corresponding to the i-th initial cluster center, Indicates the minimum value of the target probability that the j-th target edge point belongs to the target corresponding to the i-th initial cluster center.

[0116] In some implementations, the model adjustment module 205, during the process of obtaining the target position information corresponding to the target object in the initial tower crane three-dimensional model according to Beidou positioning, executes: Obtaining a target hanging object corresponding to the target tower crane from the initial three-dimensional tower crane model; Obtaining the hanging object position information corresponding to the target hanging object by using the Beidou positioning; The position of the target object in the initial tower crane three-dimensional model is calculated according to the hanging object position information to obtain the target position information corresponding to the target object.

[0117] In some implementations, the target tower crane three-dimensional model includes a first tower crane three-dimensional model corresponding to a first moment and a second tower crane three-dimensional model corresponding to a second moment adjacent to the first moment. In the process of performing real-time operation state recognition according to the target tower crane three-dimensional model to obtain the target operation state corresponding to the target tower crane, the state recognition module 206 executes: Obtain first feature information corresponding to the target hanging object from the first three-dimensional model of the tower crane and obtain second feature information corresponding to the target hanging object from the second three-dimensional model of the tower crane; Determine the change angle corresponding to each associated position in the target hanging object according to the first characteristic information and the second characteristic information; and determine the maximum movement angle corresponding to the target hanging object according to the change angle; Determine the angle difference corresponding to the associated position in the target hanging object by performing a difference calculation based on the change angle and the maximum movement angle; Determine a change weight corresponding to the associated position according to the change angle and the angle difference, and determine a target change value corresponding to the target tower crane according to the change weight, the first feature information and the second feature information; The target operating state corresponding to the target tower crane is determined according to the target change value.

[0118] In some embodiments, the intelligent tower crane remote operation system 200 based on vision and laser radar fusion recognition can be applied to terminal equipment.

[0119] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the intelligent tower crane remote operation system 200 based on vision and laser radar fusion recognition described above can refer to the corresponding process in the aforementioned embodiment of the intelligent tower crane remote operation method based on vision and laser radar fusion recognition, and will not be repeated here.

[0120] An embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of the XXXX methods provided in the description of the embodiment of the present invention.

[0121] The storage medium may be an internal storage unit of the terminal device described in the foregoing embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.

[0122] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0123] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0124] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An intelligent tower crane remote operation method based on vision and laser radar fusion recognition, characterized in that: The method comprises: Using a first laser radar at a first preset position to collect first point cloud data corresponding to the surrounding environment of the target tower crane, and using a second laser radar at a second preset position to collect second point cloud data corresponding to the surrounding environment of the target tower crane; Using a third laser radar at a third preset position to collect third point cloud data corresponding to the surrounding environment of the target tower crane, and using a visual camera at the third preset position to collect an environmental image corresponding to the surrounding environment of the target tower crane; Performing data fusion on the first point cloud data, the second point cloud data and the third point cloud data to obtain target point cloud data; Fusing the environment image and the target point cloud data to obtain an initial three-dimensional tower crane model corresponding to the surrounding environment of the target tower crane; Obtaining target position information corresponding to the target object in the initial tower crane three-dimensional model according to Beidou positioning, and adding the target position information to the initial tower crane three-dimensional model to obtain a target tower crane three-dimensional model; Performing real-time operation status recognition according to the three-dimensional model of the target tower crane to obtain a target operation status corresponding to the target tower crane; Sending the target tower crane three-dimensional model and the target operating status to a terminal device that is communicatively connected to the target tower crane, so that the terminal device displays the target tower crane three-dimensional model and the target operating status to a target user; The control instruction sent by the terminal device is obtained, and the target tower crane is controlled to perform tower crane operation according to the control instruction.

2. The method according to claim 1, characterized in that The step of fusing the first point cloud data, the second point cloud data and the third point cloud data to obtain target point cloud data includes: Performing data clustering on the first point cloud data to obtain a first clustering result and a first target type corresponding to the first clustering result; Performing data clustering on the second point cloud data to obtain a second clustering result and a second target type corresponding to the second clustering result; Performing data clustering on the third point cloud data to obtain a third clustering result and a third target type corresponding to the third clustering result; Using a target recognition model to perform target recognition on the environment image to obtain a corresponding target recognition type; Using the first target type to filter the first clustering result according to the target recognition type to obtain a first point cloud cluster; Using the second target type to filter the second clustering result according to the target recognition type to obtain a second point cloud cluster; Using the third target type to filter the third clustering result according to the target recognition type to obtain a third point cloud cluster; Performing data alignment on the first point cloud cluster and the third point cloud cluster to obtain a first alignment result, and performing data alignment on the second point cloud cluster and the third point cloud cluster to obtain a second alignment result; The target point cloud data is obtained by performing data fusion on the first point cloud data, the second point cloud data and the third point cloud data according to the first alignment result and the second alignment result.

3. The method according to claim 2, characterized in that The using the target recognition model to perform target recognition on the environment image to obtain the corresponding target recognition type includes: Using the input layer of the target recognition model to perform a preprocessing operation on the environment image to obtain a preprocessed image; Using the feature extraction layer of the target recognition model to extract features from the preprocessed image to obtain initial feature information; Using the variable convolutional layer of the target recognition model to add a learnable offset and an adjustment factor at the sampling position to adaptively perform feature sampling on the initial feature information to obtain target feature information; The target recognition layer of the target recognition model is used to perform target recognition on the target feature information to obtain the target recognition type.

4. The method according to claim 1, characterized in that: The step of fusing the environment image and the target point cloud data to obtain an initial three-dimensional tower crane model corresponding to the surrounding environment of the target tower crane includes: Performing image enhancement on the environment image to obtain a corresponding target image; Performing edge recognition on the target image to obtain target edge information corresponding to the target image, and performing target recognition on the target edge information according to a convolutional neural network to obtain a first recognition result corresponding to the target image; Performing target recognition on the target point cloud data to obtain a second recognition result corresponding to the target point cloud data; The initial tower crane three-dimensional model is obtained by performing data fusion on the environment image and the target point cloud data according to the first recognition result and the second recognition result.

5. The method according to claim 4, characterized in that The step of performing image enhancement on the environment image to obtain a corresponding target image includes: Extracting brightness features from the environment image to obtain a corresponding first brightness feature and extracting saturation features from the environment image to obtain a corresponding first saturation feature; Performing data enhancement on the first brightness feature to obtain a second brightness feature, and performing correction processing on the second brightness feature using a gamma correction algorithm to obtain a third brightness feature; Determine a target window, and obtain a window brightness feature corresponding to the third brightness feature under the target window; Performing regional histogram equalization processing on the window brightness feature to obtain a corresponding target brightness feature, and recombining according to the target brightness feature to obtain a fourth brightness feature; Performing bilateral filtering on the first saturation feature to obtain a second saturation feature, and performing stretching transformation on the second saturation feature to obtain a third saturation feature; The corresponding target image is obtained according to the fourth brightness feature and the third saturation feature.

6. The method according to claim 4, characterized in that The performing edge recognition on the target image to obtain target edge information corresponding to the target image includes: Performing wavelet transform on the target image to obtain first transform results of the target image in the horizontal direction and the vertical direction; Performing a wavelet transform on the target image to obtain a second transform result of the target image in the horizontal direction and performing a wavelet transform on the target image to obtain a third transform result of the target image in the vertical direction; Determine a target gradient corresponding to the target image according to the first transformation result, the second transformation result, and the third transformation result; Determine a segmentation threshold corresponding to the target image according to the target gradient, and determine a target edge point corresponding to the target image according to the segmentation threshold; Data clustering is performed on the target edge points to obtain target edge information corresponding to the target image.

7. The method according to claim 6, characterized in that The step of performing data clustering on the target edge points to obtain target edge information corresponding to the target image includes: Determine an initial cluster center, and obtain distance information between the target edge point and the initial cluster center; Determine, according to the distance information, the probability that the target edge point belongs to the target corresponding to the initial cluster center; Calculating a first degree of membership of the target edge point to the initial cluster center, and determining a cluster target value corresponding to the initial cluster center according to the distance information and the first degree of membership; Update the first membership degree according to the clustering target value and the target probability to obtain a second membership degree; updating the initial cluster center according to the second membership degree to obtain a target cluster center; Performing data clustering on the target edge points according to the target cluster center to obtain the target edge information corresponding to the target image; The clustering target value and the second degree of membership are obtained according to the following formula: ； ； Among them, represents the clustering target value, cer represents the number of the initial clustering centers, m represents the number corresponding to the target edge points, represents the segmentation threshold, represents the pixel value corresponding to the j-th target edge point, represents the number of pixel values ​​corresponding to the j-th target edge point, represents the first membership degree that the j-th target edge point belongs to the ith initial clustering center, represents the pixel value corresponding to the ith initial clustering center; represents the second membership degree that the j-th target edge point belongs to the ith initial clustering center, represents the target probability that the j-th target edge point belongs to the ith initial clustering center, and represents the minimum value of the target probability that the j-th target edge point belongs to the ith initial clustering center.

8. The method according to claim 1, characterized in that The step of obtaining target position information corresponding to the target object in the initial tower crane three-dimensional model according to Beidou positioning includes: Obtaining a target hanging object corresponding to the target tower crane from the initial three-dimensional tower crane model; Obtaining the hanging object position information corresponding to the target hanging object by using the Beidou positioning; The position of the target object in the initial tower crane three-dimensional model is calculated according to the hanging object position information to obtain the target position information corresponding to the target object.

9. The method according to claim 8, characterized in that The target tower crane three-dimensional model includes a first tower crane three-dimensional model corresponding to a first moment and a second tower crane three-dimensional model corresponding to a second moment adjacent to the first moment, and performing real-time operation state recognition according to the target tower crane three-dimensional model to obtain a target operation state corresponding to the target tower crane includes: Obtain first feature information corresponding to the target hanging object from the first three-dimensional model of the tower crane and obtain second feature information corresponding to the target hanging object from the second three-dimensional model of the tower crane; Determine the change angle corresponding to each associated position in the target hanging object according to the first characteristic information and the second characteristic information; and determine the maximum movement angle corresponding to the target hanging object according to the change angle; Determine the angle difference corresponding to the associated position in the target hanging object by performing a difference calculation based on the change angle and the maximum movement angle; Determine a change weight corresponding to the associated position according to the change angle and the angle difference, and determine a target change value corresponding to the target tower crane according to the change weight, the first feature information and the second feature information; The target operating state corresponding to the target tower crane is determined according to the target change value.

10. An intelligent tower crane remote control system based on vision and laser radar fusion recognition, characterized in that: include: A first acquisition module is used to acquire first point cloud data corresponding to the surrounding environment of the target tower crane using a first laser radar at a first preset position, and to acquire second point cloud data corresponding to the surrounding environment of the target tower crane using a second laser radar at a second preset position; A second acquisition module is used to collect third point cloud data corresponding to the surrounding environment of the target tower crane using a third laser radar at a third preset position and to collect an environmental image corresponding to the surrounding environment of the target tower crane using a visual camera at the third preset position; A first fusion module, used for fusing the first point cloud data, the second point cloud data and the third point cloud data to obtain target point cloud data; A second fusion module is used to fuse the environment image and the target point cloud data to obtain an initial tower crane three-dimensional model corresponding to the surrounding environment of the target tower crane; A model adjustment module, used for obtaining target position information corresponding to the target object in the initial tower crane three-dimensional model according to Beidou positioning, and adding the target position information to the initial tower crane three-dimensional model to obtain a target tower crane three-dimensional model; A state recognition module, used for performing real-time operation state recognition according to the three-dimensional model of the target tower crane to obtain a target operation state corresponding to the target tower crane; A data sending module, used for sending the target tower crane three-dimensional model and the target operating status to a terminal device that is communicatively connected to the target tower crane, so that the terminal device displays the target tower crane three-dimensional model and the target operating status to a target user; The instruction execution module is used to obtain the control instruction sent by the terminal device and control the target tower crane to perform tower crane operations according to the control instruction.

Images