Dynamic updating method, system and terminal device based on real scene three-dimensional model of tower crane operation surface

Abstract

The embodiment of the application provides a kind of based on tower crane operating surface real scene three-dimensional model dynamic updating method, system and terminal equipment, belong to tower machine operation control technical field.The method comprises: according to the feature extraction of first image information and second image information according to first focus and second focus obtains first feature information;According to the feature extraction of third image information and fourth image information according to third focus and fourth focus obtains second feature information;According to first feature information and second feature information, information fusion is carried out to obtain target feature information;According to target feature information and target point cloud data, three-dimensional reconstruction is carried out to obtain initial model;To first image information, second image information, third image information and fourth image information are carried out real-time update to obtain updated image information;And utilize updated image information to carry out dynamic detection to operating surface and obtain target detection result;According to target detection result, initial model is dynamically updated, and target model is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tower crane operation control, and particularly relates to a dynamic updating method and system of a real scene three-dimensional model based on a tower crane operation surface and a terminal device. BACKGROUND

[0002] obtain first image information corresponding to the operation surface at a first focal point of a first angle and second image information corresponding to the operation surface at a second focal point of the first angle, and obtain third image information corresponding to the operation surface at a third focal point of a second angle and fourth image information corresponding to the operation surface at a fourth focal point of the second angle, and collect target point cloud data of the operation surface by using a laser radar;

[0003] perform feature extraction on the first image information and the second image information according to the first focal point and the second focal point to obtain first feature information corresponding to the first angle;

[0004] perform feature extraction on the third image information and the fourth image information according to the third focal point and the fourth focal point to obtain second feature information corresponding to the second angle;

[0005] perform information fusion according to the first feature information and the second feature information to obtain target feature information corresponding to the operation surface;

[0006] perform three-dimensional reconstruction according to the target feature information and the target point cloud data to obtain an initial real scene three-dimensional model;

[0007] perform real-time updating on the first image information, the second image information, the third image information, and the fourth image information to obtain fifth image information corresponding to the first image information, sixth image information corresponding to the second image information, seventh image information corresponding to the third image information, and eighth image information corresponding to the fourth image information;

[0008] perform dynamic detection on the operation surface according to the fifth image information, the sixth image information, the seventh image information, and the eighth image information to obtain a target detection result;

[0009] perform dynamic updating on the initial real scene three-dimensional model according to the target detection result to obtain a target real scene three-dimensional model corresponding to the operation surface. SUMMARY

[0010] The main purpose of the embodiment of the present application is to provide a dynamic updating method and system of a real scene three-dimensional model based on a tower crane operation surface and a terminal device, and aims to solve the problem that an initially constructed three-dimensional model is difficult to comprehensively and accurately capture information, thereby leading to low efficiency of dynamic updating of the three-dimensional model.

[0011] In a first aspect, an embodiment of the present application provides a dynamic updating method of a real scene three-dimensional model based on a tower crane operation surface, comprising:

[0012] obtaining first image information corresponding to a first focal point at a first angle and second image information corresponding to a second focal point at the first angle, and obtaining third image information corresponding to a third focal point at a second angle and fourth image information corresponding to a fourth focal point at the second angle, and collecting target point cloud data of the operation surface by using a laser radar;

[0013] performing feature extraction on the first image information and the second image information according to the first focal point and the second focal point to obtain first feature information corresponding to the first angle;

[0014] performing feature extraction on the third image information and the fourth image information according to the third focal point and the fourth focal point to obtain second feature information corresponding to the second angle;

[0015] performing information fusion according to the first feature information and the second feature information to obtain target feature information corresponding to the operation surface;

[0016] performing three-dimensional reconstruction according to the target feature information and the target point cloud data to obtain an initial real scene three-dimensional model;

[0017] performing real-time updating on the first image information, the second image information, the third image information and the fourth image information to obtain fifth image information corresponding to the first image information, sixth image information corresponding to the second image information, seventh image information corresponding to the third image information and eighth image information corresponding to the fourth image information;

[0018] performing dynamic detection on the operation surface according to the fifth image information, the sixth image information, the seventh image information and the eighth image information to obtain a target detection result;

[0019] performing dynamic updating on the initial real scene three-dimensional model according to the target detection result to obtain a target real scene three-dimensional model corresponding to the operation surface.

[0020] In a second aspect, an embodiment of the present application provides a dynamic updating system of a real scene three-dimensional model based on a tower crane operation surface, comprising:

[0021] The data acquisition module is configured to obtain first image information corresponding to a first focal point at a first angle and second image information corresponding to a second focal point at the first angle of a work surface, and obtain third image information corresponding to a third focal point at a second angle and fourth image information corresponding to a fourth focal point at the second angle of the work surface, and collect target point cloud data of the work surface by using a laser radar;

[0022] The first feature extraction module is configured to perform feature extraction on the first image information and the second image information according to the first focal point and the second focal point to obtain first feature information corresponding to the first angle.

[0023] The second feature extraction module is configured to perform feature extraction on the third image information and the fourth image information according to the third focal point and the fourth focal point to obtain second feature information corresponding to the second angle.

[0024] The information fusion module is configured to perform information fusion according to the first feature information and the second feature information to obtain target feature information corresponding to the work surface.

[0025] The three-dimensional reconstruction module is configured to perform three-dimensional reconstruction according to the target feature information and the target point cloud data to obtain an initial real scene three-dimensional model.

[0026] The data updating module is configured to perform real-time updating on the first image information, the second image information, the third image information and the fourth image information to obtain fifth image information corresponding to the first image information, sixth image information corresponding to the second image information, seventh image information corresponding to the third image information and eighth image information corresponding to the fourth image information.

[0027] The dynamic detection module is configured to perform dynamic detection on the work surface according to the fifth image information, the sixth image information, the seventh image information and the eighth image information to obtain a target detection result.

[0028] The model updating module is configured to perform dynamic updating on the initial real scene three-dimensional model according to the target detection result to obtain a target real scene three-dimensional model corresponding to the work surface.

[0029] In a third aspect, the embodiments of the present application further provide a terminal device, which comprises a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection communication between the processor and the memory, wherein the computer program is executed by the processor to realize the steps of any one of the real scene three-dimensional model dynamic updating methods based on a tower crane work surface provided in the specification of the present application.

[0030] This invention provides a method, system, and terminal device for dynamically updating a real-world 3D model of a tower crane working face. The method includes: obtaining first image information corresponding to a first focal point at a first angle and second image information corresponding to a second focal point at the first angle; obtaining third image information corresponding to a third focal point at the second angle and fourth image information corresponding to a fourth focal point at the second angle; and collecting target point cloud data of the working face using LiDAR. By acquiring image information of the working face at different angles and focal points, details of the working face can be captured from multiple dimensions. Images from different angles can provide information about different sides of the working face, while images at different focal points can highlight features at different distances and levels. Therefore, feature extraction is performed on the first and second image information based on the first and second focal points to obtain first feature information corresponding to the first angle; and feature extraction is performed on the third and fourth image information based on the third and fourth focal points to obtain second feature information corresponding to the second angle. Furthermore, feature extraction is performed on image information from different angles based on different focal points to obtain first feature information corresponding to the first angle and second feature information corresponding to the second angle, respectively. This multi-angle feature extraction method can more effectively extract features of the work surface from different perspectives, avoiding mutual interference between information from different angles and improving the accuracy and effectiveness of feature extraction. Then, information fusion is performed based on the first and second feature information to obtain the target feature information corresponding to the work surface. This integrates feature information from different angles, compensating for the deficiencies of single-angle feature information, and thus providing a more comprehensive and accurate description of the work surface's features. By fusing image features from different angles, objects and structures in the work surface can be more clearly identified, improving the understanding and analysis capabilities of the work surface. Based on the target feature information and target point cloud data, a relatively accurate initial real-world 3D model can be obtained through 3D reconstruction. Furthermore, the first, second, third, and fourth image information are updated in real time to obtain the fifth, sixth, seventh, and eighth image information corresponding to the first, second, and third image information, respectively. Based on these fifth, sixth, seventh, and eighth image information, dynamic detection of the work surface is performed to obtain target detection results. Real-time image updates can promptly reflect changes in the work surface, and dynamic detection can promptly detect anomalies and trends within the work surface. The initial 3D model is then dynamically updated based on the target detection results to obtain the target 3D model corresponding to the work surface.Dynamically updating the 3D model ensures that it always reflects the latest state of the work surface, improving its timeliness and usability. Furthermore, by acquiring image information of the work surface from different angles and focal points, it is possible to capture details of the work surface from multiple dimensions, thereby effectively improving the accuracy of constructing the initial realistic 3D model. When dynamically updating the initial realistic 3D model, the model can be adjusted more efficiently based on the actual changes in the work surface. This also solves the problem in related technologies where the initially constructed 3D model is difficult to capture information comprehensively and accurately, resulting in low efficiency in dynamic updates of the 3D model. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a method for dynamically updating a real-world 3D model based on a tower crane working surface, provided in an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of the module structure of a dynamic update system for a real-scene 3D model based on a tower crane working surface, provided in an embodiment of the present invention;

[0034] Figure 3 This is a schematic block diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0037] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0038] This invention provides a method, system, and terminal device for dynamically updating a real-world 3D model based on a tower crane working surface. The method for dynamically updating the real-world 3D model based on the tower crane working surface can be applied to a terminal device, which can be an electronic device such as a tablet computer, laptop computer, desktop computer, personal digital assistant, or wearable device. The terminal device can be a server or a server cluster.

[0039] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0040] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for dynamically updating a real-world 3D model based on a tower crane working surface, as provided in an embodiment of the present invention.

[0041] like Figure 1 As shown, the method for dynamically updating the real-world 3D model based on the tower crane working surface includes steps S101 to S108.

[0042] Step S101: Obtain the first image information corresponding to the first focal point of the first angle and the second image information corresponding to the second focal point of the first angle, as well as the third image information corresponding to the third focal point of the second angle and the fourth image information corresponding to the fourth focal point of the second angle, and use the lidar to collect the target point cloud data of the work surface.

[0043] For example, when determining the working surface corresponding to the target tower crane, the appropriate use of angles and focal points plays a crucial role in the comprehensive and detailed information collection and analysis of the working surface. The organic combination of the first and second angles can effectively encompass all information about the working surface. From a spatial perspective, each working surface has multiple sides and different orientational features. Observation from a single angle often has limitations, only capturing partial information about the working surface and failing to form a complete understanding. The first and second angles, however, are two carefully selected and planned perspectives, like two observers examining the working surface from different directions, each capturing features from different orientations. For instance, in the working surface of a large construction site, the first angle might be a frontal shot, clearly showing the main structure and outline of the building; the second angle, from the side, can obtain information such as details of the side walls and ancillary facilities. Integrating the information collected from these two angles is like piecing together the pieces of a jigsaw puzzle, encompassing all information about the working surface and thus providing a comprehensive and complete understanding.

[0044] For example, the setting of the focal point is a key factor in further exploring the details of the work surface. At the first angle, the first and second focal points play a crucial role. The focal point functions similarly to the focusing function of a camera lens; by adjusting the focal point, objects at different distances and levels on the work surface can be clearly imaged. The first and second focal points can focus on different areas or objects on the work surface at the first angle, thereby determining different details of the work surface at that angle. Similarly, at the second angle, the third and fourth focal points also undertake the important task of determining different details of the work surface. They provide a more in-depth observation and analysis of the work surface based on the second angle. Thus, the combination of the first and second angles provides a macroscopic perspective for a comprehensive understanding of the work surface, while the first and second focal points, and the third and fourth focal points, focus on the work surface meticulously at different angles, enabling a comprehensive and multi-layered understanding of the work surface information from macro to micro perspectives.

[0045] For example, a first visual sensor is used to acquire first image information corresponding to the working surface at a first focal point at a first angle, and a second visual sensor is used to acquire second image information corresponding to the working surface at a second focal point at the first angle. Simultaneously, a third visual sensor is used to acquire third image information corresponding to the working surface at a third focal point at a second angle, and a fourth visual sensor is used to acquire fourth image information corresponding to the working surface at a fourth focal point at a second angle. At the same time, the acquisition path of the lidar on the working surface is determined, and information is acquired from the working surface according to the acquisition path to obtain target point cloud data of the working surface.

[0046] It should be noted that the first visual sensor, the second visual sensor, the third visual sensor, and the fourth visual sensor can be any of the image sensor or the video sensor, and this application does not impose any specific restrictions.

[0047] Step S102: Based on the first focal point and the second focal point, perform feature extraction on the first image information and the second image information to obtain the first feature information corresponding to the first angle.

[0048] For example, the specific locations and ranges of the first and second focal points in the first and second image information are analyzed. Focal points typically represent key areas of interest to the user in the image. By determining the location of the focal points, the core areas from which features need to be extracted can be clearly identified. Therefore, based on the location and range of the focal points, the corresponding regions of interest (ROIs) are extracted from the first and second image information. Shapes such as rectangles and ellipses can be used to define the ROIs, cropping them from the first and second image information respectively, reducing interference from unnecessary background information on feature extraction.

[0049] For example, based on the characteristics and requirements of the work surface, the corresponding feature type is selected, such as corner features, edge features, and texture features. Corner features can reflect the corners and abrupt changes of objects in an image, playing an important role in target recognition and matching; edge features can outline the contours of objects, aiding in shape analysis; texture features can describe the texture information of the object's surface, effectively distinguishing objects of different materials. Then, for the selected feature type, corresponding feature extraction algorithms are applied. For example, for corner features, the Harris corner detection algorithm and the Shi-Tomasi corner detection algorithm can be used; for edge features, the Canny edge detection algorithm and the Sobel operator can be used; for texture features, algorithms such as gray-level co-occurrence matrix and local binary mode (LBP) can be used. Finally, the features extracted from the regions of interest of the first and second image information are integrated to form a feature set.

[0050] For example, vectors of different features are concatenated and integrated. The integrated feature set is then filtered to remove redundant and noisy features, retaining only representative and discriminative features. These filtered features are then encoded and described, converting them into a format easy to store and process. Feature vectors, feature histograms, or other methods can be used to represent the features. The encoded and described feature information is then summarized to form the first feature information corresponding to the first angle. This first feature information should comprehensively and accurately reflect the features and information of the working surface at the first angle, providing a foundation for subsequent steps such as information fusion and 3D reconstruction.

[0051] In some embodiments, the step of extracting features from the first image information and the second image information based on the first focal point and the second focal point to obtain the first feature information corresponding to the first angle includes: performing sharpness analysis on the first image information to obtain a first blurred region corresponding to the first image information; and performing sharpness analysis on the second image information to obtain a second blurred region corresponding to the second image information; determining the adjacent range between the first image information and the second image information at the same pixel point, the adjacent range including adjacent length and adjacent width; determining the first blurred region corresponding to the first blurred region in the second image information based on the first blurred region combined with the adjacent length and the adjacent width; and determining the second blurred region in the second image information based on the second blurred region combined with the adjacent length and the adjacent width. The image information includes: a second initial clear region corresponding to the first blurred region; data alignment based on the first blurred region and the first initial clear region to obtain a first target clear region corresponding to the first blurred region in the second image information; data alignment based on the second blurred region and the second initial clear region to obtain a second target clear region corresponding to the second blurred region in the first image information; image fusion based on the first target clear region and the first image information to obtain first fusion information; image fusion based on the second target clear region and the second image information to obtain second fusion information; determining a first fused image corresponding to the working surface at the first angle based on the first fusion information and the second fusion information; and feature extraction based on the first fused image to obtain the first feature information corresponding to the working surface at the first angle.

[0052] For example, a sharpness assessment algorithm such as the Laplacian variance method is used to perform a Laplacian transform on the first image information, and the variance of the transformed image is calculated. The larger the variance, the sharper the image. Thus, the first blurred region corresponding to the first image information is obtained based on the variance of the transformed image. For instance, a suitable sharpness threshold is determined based on actual conditions and experience. The sharpness index of each small block in the first image information is compared with this threshold; if it is lower than the threshold, the small block is marked as a blurred region. Finally, all marked blurred blocks are integrated to obtain the first blurred region corresponding to the first image information. Similarly, a Laplacian transform is performed on the second image information, and the variance of the transformed image is calculated. Thus, the second blurred region corresponding to the second image information is obtained based on the variance of the transformed image.

[0053] For example, the adjacent length and adjacent width are determined by comprehensively considering the characteristics of the first image information and the second image information, as well as the application scenario. For instance, if the size of the object in the image is large, the adjacent length and adjacent width can be set to be larger; if more refined processing is required, the adjacent length and adjacent width can be set to be smaller.

[0054] For example, for each pixel or pixel block within the first blurred region (the appropriate granularity can be selected according to the actual situation), its position in the first image information is used as a reference. Based on the previously determined adjacent length and adjacent width, the corresponding adjacent region is found in the second image information. Specifically, taking the position of the pixel or pixel block in the first blurred region as the center, the adjacent length and adjacent width are extended horizontally and vertically in the second image, respectively, to determine a rectangular adjacent region. Then, the sharpness of each adjacent region found in the second image is evaluated using the method previously used to analyze image sharpness, such as the Laplacian variance method, to calculate its sharpness index. A sharpness threshold is set, and the sharpness index of the adjacent regions is compared with the threshold. If the sharpness index of a certain adjacent region is higher than the threshold, it is marked as a sharp region. Then, all the adjacent regions marked as sharp are integrated to obtain the first initial sharp region corresponding to the first blurred region in the second image information.

[0055] For example, similar to determining the first initial sharp region, each pixel or pixel block within the second blurred region is traversed. Based on adjacent length and adjacent width, corresponding neighboring regions are found in the first image information. Using the position of the pixel or pixel block in the second blurred region as the center, corresponding distances are extended horizontally and vertically to determine rectangular neighboring regions. Using the same sharpness evaluation method, the sharpness of each neighboring region found in the first image information is calculated to obtain its sharpness index. Using the same sharpness threshold as when determining the first initial sharp region, the sharpness index of the neighboring regions is compared with this threshold. Neighboring regions with values ​​higher than the threshold are marked as sharp regions. All marked sharp neighboring regions are combined to obtain the second initial sharp region corresponding to the second blurred region in the first image information.

[0056] For example, feature points are extracted from the first blurred region and the first initially sharp region using feature extraction algorithms such as SIFT, SURF, or ORB. These extracted feature points are then matched by comparing the similarity of feature descriptors to find corresponding feature point pairs in the two regions. Based on the matched feature point pairs, a transformation matrix that can transform the first initially sharp region to align with the first blurred region is calculated using methods such as least squares. This matrix describes the geometric transformation relationship between the two regions. Using the calculated transformation matrix, the first initially sharp region is geometrically transformed, adjusting its position and orientation to match the first blurred region, thus obtaining the first target sharp region corresponding to the first blurred region in the second image information.

[0057] Following the same steps of aligning the first blurred region with the first initial clear region, feature points are extracted and matched in the second blurred region and the second initial clear region. The transformation matrix is ​​calculated and the transformation is performed to finally obtain the second target clear region corresponding to the second blurred region in the first image information.

[0058] For example, a multi-resolution fusion method, such as Laplacian pyramid fusion or wavelet fusion, is used to fuse the first sharp target region and the first image information to obtain first fused information. Then, the same method is used to fuse the second sharp target region and the second image information to obtain second fused information.

[0059] For example, although the blurred region and the initial sharp region have been aligned previously, image registration is required again to ensure that the first fusion information and the second fusion information are accurately matched in position and scale. The same feature matching and transformation matrix calculation methods as before can be used to register the first fusion information and the second fusion information, and then the registered first fusion information and the second fusion information are fused to obtain the first fused image corresponding to the working surface at the first angle.

[0060] For example, based on the characteristics of the work surface and the analysis requirements, a corresponding feature extraction algorithm, such as the SIFT algorithm, is selected to extract feature points and corresponding feature descriptors from the first fused image. These feature points and descriptors together constitute the first feature information corresponding to the work surface at the first angle.

[0061] In some embodiments, the step of obtaining a first target clear region corresponding to the first blurred region in the second image information by data alignment based on the first blurred region and the first initial clear region, and obtaining a second target clear region corresponding to the second blurred region in the first image information by data alignment based on the second blurred region and the second initial clear region, includes: calculating a first mapping difference when the first blurred region corresponds to a feature in the first initial clear region by performing loss calculation using the adjacent length and the adjacent width based on the first initial clear region and the first blurred region; calculating a second mapping difference when the second blurred region corresponds to a feature in the second initial clear region by performing loss calculation using the adjacent length and the adjacent width based on the second initial clear region and the second blurred region; determining a first target clear region corresponding to the first blurred region in the second image information based on the first mapping difference and the first initial clear region; and determining a second target clear region corresponding to the second blurred region in the first image information based on the second mapping difference and the second initial clear region.

[0062] For example, based on the previously determined adjacent length and adjacent width, the first initially sharp region and the first blurred region are divided into multiple sub-regions of the same size (determined by the adjacent length and adjacent width). It is ensured that sub-regions at the same location in the two regions have a spatial correspondence. For each sub-region, features are extracted. Features can be image color features (such as RGB mean, variance, etc.), texture features (such as gray-level co-occurrence matrix features), or higher-level features (such as SIFT, HOG, etc. feature descriptors). For corresponding sub-regions in the first initially sharp region and the first blurred region, the feature differences between them are calculated. Common loss calculation methods include Euclidean distance, Manhattan distance, cosine similarity, etc.

[0063] For example, if Euclidean distance is used to calculate the difference in color features, the Euclidean distance between the RGB mean vectors of the corresponding sub-regions is calculated as the loss value for that sub-region. The loss values ​​of all corresponding sub-regions are then summed or weighted, and the loss values ​​are optimized to obtain the first mapping difference corresponding to the coordinate mapping of the first blurred region in the first initial sharp region. That is, the first mapping difference is the coordinate difference corresponding to the coordinate information of the first blurred region when mapped in the second image information.

[0064] For example, the first initial clear region is adjusted or transferred according to the first mapping difference to determine the first target clear region corresponding to the first blurred region in the second image information.

[0065] For example, the second initial sharp region and the second blurred region are divided into multiple sub-regions of the same size according to adjacent length and adjacent width, and a correspondence is established. Features are extracted from each sub-region, and the feature type can be consistent with that used when calculating the first mapping difference. Using the same loss calculation method as for calculating the first mapping difference, the feature differences between corresponding sub-regions in the second initial sharp region and the second blurred region are calculated. The loss values ​​of each corresponding sub-region are summed or weighted to obtain the second mapping difference corresponding to the features of the second blurred region in the second initial sharp region. That is, the second mapping difference is the coordinate difference corresponding to the coordinate information of the second blurred region when mapped in the first image information.

[0066] For example, the second initial clear region is adjusted or shifted according to the second mapping difference to determine the second target clear region corresponding to the second blurred region in the first image information.

[0067] In some embodiments, obtaining the first mapping difference when the first blurred region corresponds to a feature in the first initial clear region by performing loss calculation based on the first initial clear region and the first blurred region using the adjacent length and the adjacent width includes: obtaining a first pixel value corresponding to the first blurred region at a first position, and determining first range information corresponding to the first initial clear region based on the first position, the adjacent length, and the adjacent width; obtaining a second pixel value corresponding to the first initial clear region under the first range information; constructing a first loss function between the first initial clear region and the first blurred region based on the first pixel value and the second pixel value; obtaining a first derivative function by taking the derivative of the first loss function, and obtaining a second derivative function by performing Taylor expansion on the first derivative function; obtaining the first mapping difference when the first blurred region corresponds to a feature in the first initial clear region by performing displacement assumption processing on the second derivative function; wherein, the first mapping difference is obtained according to the following formula:

[0068] ;

[0069] in, The first mapping difference represents the feature correspondence in the first initially clear region when the horizontal position is x and the vertical position is y in the first blurred region; w represents the adjacent length; and h represents the adjacent width. This represents the value of the second derivative function when the horizontal position corresponding to the first blurred region is i. This represents the value of the second derivative function when the vertical position corresponding to the first blurred region is j. This represents the first pixel value corresponding to a horizontal position of i and a vertical position of j in the first blurred region. This represents the second pixel value corresponding to the horizontal position i and the vertical position j in the first initial clear region.

[0070] For example, the first pixel value corresponding to any first position in the first blurred region is obtained, and then the region is expanded horizontally and vertically with the first position as the center, based on the adjacent length and adjacent width. In the horizontal direction, the adjacent length is expanded to the left and right by half from the first position, respectively; in the vertical direction, the adjacent width is expanded upward and downward by half from the first position, respectively. The region enclosed by the expanded boundary is the first range information corresponding to the first initial clear region, which is also a rectangular region, represented by the pixel coordinates of the upper left and lower right corners of the region.

[0071] For example, within a first initially clear region, the second pixel values ​​of all pixels within the region determined by the first range information are extracted. Then, the first and second pixel values ​​are substituted into a selected loss metric formula according to a loss metric such as mean squared error (MSE) or absolute error (MAE) to construct a first loss function. For instance, if mean squared error is used, the first loss function is the average of the sum of the squares of the differences between each corresponding pixel value in the first and second pixel values.

[0072] For example, the derivative is calculated using mathematical differentiation rules based on the specific form of the first loss function. If the loss function is a complex function, the chain rule, product rule, etc., may be required. Differentiating the variables in the first loss function, such as the horizontal and vertical positions, yields the first derivative function. Then, using a second-order Taylor expansion, the first derivative function is expanded at a suitable point (usually the current estimate), ignoring higher-order infinitesimal terms, to obtain the second derivative function.

[0073] For example, suppose there is a displacement relationship between the first blurred region and the first initially sharp region. This displacement can be represented by a vector, such as a two-dimensional vector in a two-dimensional image, representing the displacement in the horizontal and vertical directions. Taking the displacement relationship as small, the displacement assumed in the displacement relationship is substituted into the second derivative function. By solving the function, the first mapping difference when the features of the first blurred region correspond to those in the first initially sharp region is obtained. The first mapping difference is the coordinate difference when the coordinate information corresponding to the first blurred region is mapped in the second image information. The first mapping difference is obtained according to the following formula:

[0074] ;

[0075] in, This represents the first mapping difference when the feature in the first initially sharp region corresponds to the horizontal position x and the vertical position y in the first blurred region. w represents the adjacent length and h represents the adjacent width. This represents the value of the second derivative function when the horizontal position is i corresponding to the first fuzzy region. This represents the value of the second derivative function when the vertical position is j corresponding to the first fuzzy region. This represents the first pixel value corresponding to a horizontal position of i and a vertical position of j within the first blurred region. This represents the second pixel value corresponding to the horizontal position i and the vertical position j in the first initially clear region.

[0076] For example, the first mapping difference comprehensively considers pixel information within adjacent length and width ranges, as well as the value of the second derivative function. This allows for a more comprehensive and accurate measurement of the feature correspondence between the first blurred region and the first initially sharp region during image registration. By calculating this difference, the best-matching position between the two regions can be found, thereby achieving more accurate image registration, reducing registration errors, and making the registered images more spatially aligned.

[0077] Step S103: Based on the third focal point and the fourth focal point, perform feature extraction on the third image information and the fourth image information to obtain the second feature information corresponding to the second angle.

[0078] For example, the same technical means as "extracting features from the first image information and the second image information based on the first focal point and the second focal point to obtain the first feature information corresponding to the first angle" is used to extract features from the third image information and the fourth image information based on the third focal point and the fourth focal point to obtain the second feature information corresponding to the second angle. This application will not elaborate further.

[0079] Step S104: Perform information fusion based on the first feature information and the second feature information to obtain the target feature information corresponding to the working surface.

[0080] For example, similar features are obtained by identifying similar features based on the first feature information and the second feature information. Then, the first feature information is removed to obtain the target similar features and the second feature information is removed to obtain the target similar features and the fourth feature information. The third feature information, the target similar features and the fourth feature information are then spliced ​​together to obtain the target feature information corresponding to the working surface.

[0081] In some embodiments, obtaining target feature information corresponding to the work surface by fusing information based on the first feature information and the second feature information includes: calculating similarity based on the first feature information and the second feature information to obtain a target similarity value; obtaining a target common region between the first image information and the second image information based on the target similarity value; fusing the first image information and the second image information based on the target common region to obtain an initial fused image; performing target deformation detection on the initial fused image to obtain a deformation detection result, and correcting the initial fused image based on the deformation detection result to obtain a target fused image; and performing feature extraction on the target fused image to obtain the target feature information corresponding to the work surface.

[0082] For example, target segmentation is performed on the first feature information to obtain first sub-information corresponding to multiple first sub-targets, and target segmentation is performed on the second feature information to obtain second sub-information corresponding to multiple second sub-targets. Then, Euclidean distance or Manhattan distance is used to calculate the similarity between the first sub-information and the second sub-information to obtain the target similarity value.

[0083] For example, a similarity threshold is set according to the actual situation. When the target similarity value is greater than the threshold, the corresponding first sub-information and second sub-information are considered similar, thereby finding similar regions in the first image information and second image information based on the target similarity value. These regions are the target common regions.

[0084] For example, within the target shared area, the first image information and the second image information are fused using methods such as Laplacian pyramid fusion or wavelet transform fusion; in the non-shared area, information from a certain image can be selected to be retained according to specific needs, and then the fused information corresponding to the target shared area and the retained information corresponding to the non-shared area are stitched together to obtain the initial fused image.

[0085] For example, feature point matching is used to determine the presence of deformation by detecting changes in the positions of feature points in the initial fused image, thus obtaining a deformation detection result. The initial fused image is then corrected based on the deformation detection result. If local deformation is detected, image transformations (such as affine transformations and perspective transformations) can be used to correct the deformed area; if the deformation is more complex, a machine learning model may be needed for more precise correction. Finally, the target fused image is obtained.

[0086] For example, feature extraction methods, including but not limited to color feature extraction, texture feature extraction, and shape feature extraction, are used to extract features from the target fusion image to obtain target feature information corresponding to the working surface.

[0087] In some embodiments, obtaining deformation detection results by performing target deformation detection on the initial fused image includes: obtaining a first sub-image corresponding to the target common region from the first image information and obtaining a second sub-image corresponding to the target common region from the second image information; obtaining a third sub-image corresponding to the target common region after fusion from the target fused image; calculating a first standard deviation corresponding to the first sub-image, a second standard deviation corresponding to the second sub-image, and a third standard deviation corresponding to the third sub-image; calculating a first mean corresponding to the first sub-image, a second mean corresponding to the second sub-image, and a third mean corresponding to the third sub-image; and according to the... A first correlation value is obtained between the first sub-image and the third sub-image by calculating the correlation value using the first standard deviation, the third standard deviation, the first mean, and the third mean combined with the first sub-image and the third sub-image; a second correlation value is obtained between the second sub-image and the third sub-image by calculating the correlation value using the second standard deviation, the third standard deviation, the second mean, and the third mean combined with the second sub-image and the third sub-image; the first correlation value and the second correlation value are fused to obtain a target fusion value corresponding to the initial fused image; and target deformation detection is performed on the initial fused image based on the target fusion value to obtain the deformation detection result.

[0088] For example, based on the location information of the common area of ​​the target, a corresponding first sub-image is cropped from the first image information, a corresponding second sub-image is cropped from the second image information, and a corresponding third sub-image is cropped from the target fused image.

[0089] For example, for the first sub-image, all its pixel values ​​are iterated through, and the first standard deviation is calculated according to the standard deviation calculation formula (first calculate the average of the pixel values, then calculate the average of the sum of squares of the differences between each pixel value and the average, and finally take the square root). Similarly, the second standard deviation of the second sub-image and the third standard deviation of the third sub-image are calculated using the same steps.

[0090] For example, for the first sub-image, the first mean is obtained by summing all pixel values ​​and dividing by the total number of pixels. The second mean of the second sub-image and the third mean of the third sub-image are calculated using the same method.

[0091] For example, first pixels and third pixels at the same coordinates are obtained from the first sub-image and the third sub-image, respectively. Then, the first pixel is subtracted from its first mean and divided by its first standard deviation to obtain first data, and the third pixel is subtracted from its third mean and divided by its third standard deviation to obtain third data. The first data is then subtracted from the third data and squared to obtain first squared data. Finally, the sum of all first squared data yields the first correlation value between the first sub-image and the third sub-image. The first correlation value reflects the degree of similarity and correlation between the first sub-image and the third sub-image.

[0092] For example, second and third pixels at the same coordinates are obtained from the second and third sub-images, respectively. Then, the second pixel is subtracted from the second mean and divided by the second standard deviation to obtain second data, and the third pixel is subtracted from the third mean and divided by the third standard deviation to obtain third data. The second data is then subtracted from the third data and squared to obtain second squared data. Finally, the sum of all second squared data is used to obtain the second correlation value between the second and third sub-images. The second correlation value reflects the degree of similarity and correlation between the second and third sub-images.

[0093] For example, a weighted average method is used to fuse the first and second correlation values. Appropriate weights are assigned to the first and second correlation values ​​based on their importance or reliability, and then the first and second correlation values ​​are summed according to their assigned weights to obtain the target fused value corresponding to the initial fused image.

[0094] For example, a deformation threshold is set based on actual conditions and experience. This threshold is used to determine whether the initial fused image has deformation, thereby comparing the target fused value with the set deformation threshold. If the target fused value is lower than the threshold, it indicates that the deformation detection result is that the initial fused image may have deformation; if the target fused value is higher than the threshold, it indicates that the deformation detection result is that the initial fused image has no obvious deformation.

[0095] Step S105: Perform three-dimensional reconstruction based on the target feature information and the target point cloud data to obtain an initial real-world three-dimensional model.

[0096] For example, representative features relevant to 3D reconstruction are selected from the target feature information, while redundant or noisy features are removed. For instance, if the feature information contains multiple attributes, only features related to object shape, position, texture, etc., that are helpful for 3D reconstruction are retained. Statistical filtering, radius filtering, and other algorithms are then used to remove outliers and noise points from the target point cloud data, improving the quality of the point cloud data and obtaining updated target point cloud data.

[0097] For example, the correspondence between target feature information and target point cloud data can be found. Matching can be performed based on attributes such as the geometric position and color of the features. For instance, for an object with obvious feature points, the corresponding points in the point cloud data can be found using the position information of the feature points. Based on this matching, a mapping relationship between target feature information and target point cloud data can be established, so that each feature can correspond to a specific region or set of points in the point cloud data. Then, a triangular mesh can be constructed using the associated point cloud data. The Delaunay triangulation algorithm can be used to connect the points in the point cloud data into triangles, forming an initial 3D surface model.

[0098] In some embodiments, the step of obtaining an initial real-world 3D model by performing 3D reconstruction based on the target feature information and the target point cloud data includes: obtaining target edge information corresponding to a target object in the work surface based on the target feature information; obtaining target normal information and target curvature information corresponding to the target object in the work surface based on the target point cloud data; determining a target mapping relationship between the target point cloud data and the target fused image based on the target edge information, the target normal information, and the target curvature information; calculating distance information between the target point cloud data and the target edge information based on the target mapping relationship; obtaining associated point cloud data corresponding to the target edge information from the target point cloud data based on the distance information; and performing 3D reconstruction based on the associated point cloud data and the target edge information to obtain the initial real-world 3D model corresponding to the work surface.

[0099] For example, edge detection algorithms such as the Canny operator and the Sobel operator are used to identify the pixel positions of the target object's edges by recognizing the target's feature information, thereby obtaining the target edge information.

[0100] For example, the target point cloud data is preprocessed, including denoising and downsampling, to obtain updated target point cloud data. Then, using information about each point and its neighboring points in the target point cloud data, the normal direction of each point on the target object's surface is calculated using methods such as least squares plane fitting, thus obtaining the target normal information. Based on the target normal information and the local geometry of the target point cloud data, the curvature of the target object's surface is calculated. For example, the magnitude of the curvature is determined by analyzing the rate of change of the point cloud data in a local region, thereby obtaining the target curvature information.

[0101] For example, target edge information, target normal information, and target curvature information are correlated and analyzed to find the inherent relationships between them. For instance, the normal and curvature at the target edge may have specific variation patterns. Based on the above correlation, a correspondence is established between the target point cloud data and the target fused image. Then, a feature matching algorithm is used to match the target edge information in the target fused image with the points in the target point cloud data, thereby determining the target mapping relationship.

[0102] For example, for each point in the target point cloud data, the distance to the edge position represented by the target edge information is calculated using metrics such as Euclidean distance to obtain the distance information between the target point cloud data and the target edge information. Based on the calculated distance information, an appropriate distance threshold is set. Points whose distance to the target edge information is less than this threshold are considered to be associated with the target edge information. Points that meet the above conditions are then selected from the target point cloud data, and these points constitute the associated point cloud data corresponding to the target edge information.

[0103] For example, a 3D model is constructed based on triangulation using associated point cloud data and target edge information. During the construction process, the target edge information is fully utilized to constrain the model's boundaries, ensuring that the reconstructed model accurately reflects the shape of the target object in the work area, ultimately obtaining the initial real-world 3D model corresponding to the work area.

[0104] Step S106: Update the first image information, the second image information, the third image information, and the fourth image information in real time to obtain the fifth image information corresponding to the first image information, the sixth image information corresponding to the second image information, the seventh image information corresponding to the third image information, and the eighth image information corresponding to the fourth image information.

[0105] For example, a first image information is obtained by using a first vision sensor to update the first image information corresponding to the first focal point at the first angle in real time. A sixth image information is obtained by using a second vision sensor to update the second image information corresponding to the second focal point at the first angle in real time. A seventh image information is obtained by using a third vision sensor to update the third image information corresponding to the third focal point at the second angle in real time. An eighth image information is obtained by using a fourth vision sensor to update the fourth image information corresponding to the fourth focal point at the second angle in real time.

[0106] Step S107: Perform dynamic detection on the work surface based on the fifth image information, the sixth image information, the seventh image information, and the eighth image information to obtain the target detection result.

[0107] For example, a first correlation value is obtained by performing similarity calculation on the first image information and the fifth image information, a second correlation value is obtained by performing similarity calculation on the second image information and the sixth image information, a third correlation value is obtained by performing similarity calculation on the third image information and the seventh image information, and a fourth correlation value is obtained by performing similarity calculation on the fourth image information and the eighth image information.

[0108] For example, the minimum value among the first correlation value, the second correlation value, the third correlation value and the fourth correlation value is obtained, and the minimum correlation value is compared with a preset value. When the minimum correlation value is less than the preset value, the target detection result corresponding to the work surface is determined to be that the work surface has changed; when the minimum correlation value is greater than or equal to the preset value, the target detection result corresponding to the work surface is determined to be that the work surface has not changed.

[0109] In some embodiments, the step of dynamically detecting the work surface based on the fifth image information, the sixth image information, the seventh image information, and the eighth image information to obtain target detection results includes: performing target recognition on the first image information to obtain first target information, and performing target recognition on the fifth image information to obtain fifth target information; calculating the first curvature information corresponding to the first target information when performing target segmentation and the fifth curvature information corresponding to the fifth target information when performing target segmentation; performing difference calculation based on the first curvature information and the fifth curvature information to obtain first difference information; performing target recognition on the second image information to obtain second target information, and performing target recognition on the sixth image information to obtain sixth target information; calculating the second curvature information corresponding to the second target information when performing target segmentation and the sixth curvature information corresponding to the sixth target information when performing target segmentation; and performing difference calculation based on the second curvature information and the sixth curvature information to obtain the first target information. The process involves: 1) Performing target recognition on the third image information to obtain third target information, and performing target recognition on the seventh image information to obtain seventh target information; 2) Calculating the third curvature information corresponding to the target segmentation of the third target information and the seventh curvature information corresponding to the target segmentation of the seventh target information; 3) Performing difference calculation based on the third curvature information and the seventh curvature information to obtain third difference information; 4) Performing target recognition on the fourth image information to obtain fourth target information, and performing target recognition on the eighth image information to obtain eighth target information; 5) Calculating the fourth curvature information corresponding to the target segmentation of the fourth target information and the eighth curvature information corresponding to the target segmentation of the eighth target information; 6) Performing difference calculation based on the fourth curvature information and the eighth curvature information to obtain fourth difference information; 7) Fusing the first difference information, the second difference information, the third difference information, and the fourth difference information to dynamically detect the working surface and obtain the target detection result.

[0110] For example, a deep learning-based object detection model such as YOLO (You Only Look Once) is used to process the first and fifth image information respectively to identify the target objects, thereby obtaining the first and fifth target information. Similarly, target recognition is performed sequentially on the second and sixth, third and seventh, and fourth and eighth image information to obtain the second and sixth, third and seventh, and fourth and eighth target information respectively.

[0111] For example, image segmentation algorithms, such as threshold-based segmentation and edge-based segmentation, are used to perform target segmentation operations on each target information to separate the target object from the background. Then, for the segmented results of the first and fifth target information, the curvature information corresponding to each target segmentation is calculated to obtain the first curvature information and the fifth curvature information, respectively. Curvature is a quantity that describes the degree of bending of a curve or surface, and can be represented by calculating the curvature of each point on the target boundary. Similarly, the second and sixth curvature information corresponding to the second and sixth target information, the third and seventh curvature information corresponding to the third and seventh target information, and the fourth and eighth curvature information corresponding to the fourth and eighth target information are calculated, respectively.

[0112] For example, the difference between the first curvature information and the fifth curvature information is calculated to obtain the first difference information. The difference calculation can be the difference between the curvature values ​​of corresponding points, or it can be a measure of the difference in the overall curvature distribution characteristics, such as the difference between the mean and variance. Following the same method, the second difference information between the second curvature information and the sixth curvature information, the third difference information between the third curvature information and the seventh curvature information, and the fourth difference information between the fourth curvature information and the eighth curvature information are calculated respectively.

[0113] For example, the maximum value among the first difference information, the second difference information, the third difference information, and the fourth difference information is obtained, and then the maximum difference information is compared with a preset difference value. When the maximum difference information is greater than or equal to the preset difference value, the target detection result corresponding to the working surface is determined to be that the working surface has changed; when the maximum difference information is less than the preset difference value, the target detection result corresponding to the working surface is determined to be that the working surface has not changed.

[0114] Step S108: Dynamically update the initial real-scene 3D model based on the target detection results to obtain the target real-scene 3D model corresponding to the work surface.

[0115] For example, when the target detection result is that the working surface has not changed, there is no need to dynamically update the initial real scene 3D model, that is, the target real scene 3D model is the same as the initial real scene 3D model.

[0116] For example, when the target detection result indicates that the work surface has changed, it is necessary to obtain point cloud data at the same time as the fifth image information acquisition time. Then, based on the point cloud data at the same time, the initial real scene 3D model is updated in combination with the fifth image information, the sixth image information, the seventh image information, and the eighth image information to obtain the target real scene 3D model corresponding to the work surface.

[0117] Please see Figure 2 , Figure 2This application provides a dynamic update system 200 for a real-scene 3D model of a tower crane working face. The system includes a data acquisition module 201, a first feature extraction module 202, a second feature extraction module 203, an information fusion module 204, a 3D reconstruction module 205, a data update module 206, a dynamic detection module 207, and a model update module 208. The data acquisition module 201 is used to obtain first image information corresponding to a first focal point at a first angle and second image information corresponding to a second focal point at a first angle, as well as third image information corresponding to a third focal point at a second angle and fourth image information corresponding to a fourth focal point at a second angle, and to acquire target point cloud data of the working face using a lidar. The first feature extraction module 202 is used to extract features from the first image information and the second image information based on the first focal point and the second focal point to obtain first feature information corresponding to the first angle. The second feature extraction module 203 is used to extract features from the third image information and the second image information based on the third focal point and the fourth focal point. The fourth image information is used to extract features to obtain the second feature information corresponding to the second angle; the information fusion module 204 is used to perform information fusion based on the first feature information and the second feature information to obtain the target feature information corresponding to the work surface; the three-dimensional reconstruction module 205 is used to perform three-dimensional reconstruction based on the target feature information and the target point cloud data to obtain an initial real-scene three-dimensional model; the data update module 206 is used to update the first image information, the second image information, the third image information and the fourth image information in real time to obtain the fifth image information corresponding to the first image information, the sixth image information corresponding to the second image information, the seventh image information corresponding to the third image information and the eighth image information corresponding to the fourth image information; the dynamic detection module 207 is used to perform dynamic detection on the work surface based on the fifth image information, the sixth image information, the seventh image information and the eighth image information to obtain the target detection result; the model update module 208 is used to dynamically update the initial real-scene three-dimensional model based on the target detection result to obtain the target real-scene three-dimensional model corresponding to the work surface.

[0118] In some implementations, the dynamic update system 200 based on the real-world 3D model of the tower crane working surface can be applied to terminal devices.

[0119] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the above-described dynamic update system 200 for a real-world 3D model based on a tower crane working surface can be referred to the corresponding process in the aforementioned embodiment of the dynamic update method for a real-world 3D model based on a tower crane working surface, and will not be repeated here.

[0120] Please see Figure 3 , Figure 3 This is a schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0121] like Figure 3 As shown, the terminal device 300 includes a processor 301 and a memory 302, which are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0122] Specifically, processor 301 provides computing and control capabilities to support the operation of the entire terminal device. Processor 301 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0123] Specifically, the memory 302 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0124] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the embodiments of the present invention, and does not constitute a limitation on the terminal device to which the embodiments of the present invention are applied. A specific server may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0125] The processor is used to run a computer program stored in a memory, and when executing the computer program, implements any of the methods for dynamically updating a real-world 3D model based on a tower crane working face provided in the embodiments of the present invention.

[0126] In one embodiment, the processor is configured to run a computer program stored in memory, and when executing the computer program, perform the following steps:

[0127] The system obtains first image information corresponding to the first focal point of the first angle and second image information corresponding to the second focal point of the first angle, as well as third image information corresponding to the third focal point of the second angle and fourth image information corresponding to the fourth focal point of the second angle, and collects target point cloud data of the working surface using lidar.

[0128] Based on the first focal point and the second focal point, feature extraction is performed on the first image information and the second image information to obtain the first feature information corresponding to the first angle;

[0129] Based on the third focal point and the fourth focal point, feature extraction is performed on the third image information and the fourth image information to obtain the second feature information corresponding to the second angle;

[0130] The target feature information corresponding to the work surface is obtained by information fusion based on the first feature information and the second feature information;

[0131] An initial real-scene 3D model is obtained by performing 3D reconstruction based on the target feature information and the target point cloud data;

[0132] The first image information, the second image information, the third image information, and the fourth image information are updated in real time to obtain the fifth image information corresponding to the first image information, the sixth image information corresponding to the second image information, the seventh image information corresponding to the third image information, and the eighth image information corresponding to the fourth image information;

[0133] The target detection result is obtained by dynamically detecting the working surface based on the fifth image information, the sixth image information, the seventh image information, and the eighth image information;

[0134] The initial real-scene 3D model is dynamically updated based on the target detection results to obtain the target real-scene 3D model corresponding to the work surface.

[0135] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the terminal device described above can be referred to the corresponding process in the aforementioned embodiment of the method for dynamically updating the real-scene 3D model based on the tower crane working surface, and will not be repeated here.

[0136] This invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement the steps of any of the methods for dynamically updating a real-world 3D model based on a tower crane working face as provided in the specification of this invention.

[0137] The storage medium can be an internal storage unit of the terminal device described in the foregoing embodiments, such as the hard drive or memory of the terminal device. Alternatively, the storage medium can be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device.

[0138] Those skilled in the art will understand that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0139] It should be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0140] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The above descriptions are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dynamic updating method of a real scene three-dimensional model based on a tower crane operation surface, characterized in that, The method comprises: obtaining first image information corresponding to a first focal point of a working surface at a first angle and second image information corresponding to a second focal point of the working surface at the first angle, and obtaining third image information corresponding to a third focal point of the working surface at a second angle and fourth image information corresponding to a fourth focal point of the working surface at the second angle, and collecting target point cloud data of the working surface by using a laser radar; performing feature extraction on the first image information and the second image information according to the first focal point and the second focal point to obtain first feature information corresponding to the first angle, comprising: performing sharpness analysis on the first image information to obtain a first fuzzy area corresponding to the first image information; and performing sharpness analysis on the second image information to obtain a second fuzzy area corresponding to the second image information; determining a neighboring range between the same pixel points between the first image information and the second image information, the neighboring range comprising a neighboring length and a neighboring width; determining a first initial clear area corresponding to the first fuzzy area in the second image information according to the first fuzzy area in combination with the neighboring length and the neighboring width; determining a second initial clear area corresponding to the second fuzzy area in the first image information according to the second fuzzy area in combination with the neighboring length and the neighboring width; performing data alignment according to the first fuzzy area and the first initial clear area to obtain a first target clear area corresponding to the first fuzzy area in the second image information, and performing data alignment according to the second fuzzy area and the second initial clear area to obtain a second target clear area corresponding to the second fuzzy area in the first image information, comprising: performing loss calculation on the first initial clear area and the first fuzzy area by using the neighboring length and the neighboring width to obtain a first mapping difference value when the features of the first fuzzy area correspond to the first initial clear area; performing loss calculation on the second initial clear area and the second fuzzy area by using the neighboring length and the neighboring width to obtain a second mapping difference value when the features of the second fuzzy area correspond to the second initial clear area; determining the first target clear area corresponding to the first fuzzy area in the second image information according to the first mapping difference value and the first initial clear area; determining the second target clear area corresponding to the second fuzzy area in the first image information according to the second mapping difference value and the second initial clear area; performing image fusion according to the first target clear area and the first image information to obtain first fusion information; performing image fusion according to the second target clear area and the second image information to obtain second fusion information; determining a first fusion image corresponding to the working surface at the first angle according to the first fusion information and the second fusion information; performing feature extraction on the first fusion image to obtain first feature information corresponding to the working surface at the first angle; performing feature extraction on the third image information and the fourth image information according to the third focal point and the fourth focal point to obtain second feature information corresponding to the second angle; performing information fusion according to the first feature information and the second feature information to obtain target feature information corresponding to the working surface; performing three-dimensional reconstruction according to the target feature information and the target point cloud data to obtain an initial real scene three-dimensional model; and performing three-dimensional reconstruction according to the target feature information and the target point cloud data to obtain an initial real scene three-dimensional model. The first image information, the second image information, the third image information and the fourth image information are updated in real time to obtain fifth image information corresponding to the first image information, sixth image information corresponding to the second image information, seventh image information corresponding to the third image information and eighth image information corresponding to the fourth image information; The work surface is dynamically detected according to the fifth image information, the sixth image information, the seventh image information and the eighth image information to obtain a target detection result; The initial real scene three-dimensional model is dynamically updated according to the target detection result to obtain a target real scene three-dimensional model corresponding to the work surface.

2. The method of claim 1, wherein, The first mapping difference value of the first fuzzy area when the features in the first fuzzy area correspond to the first initial clear area is obtained by using the adjacent length and the adjacent width according to the first initial clear area and the first fuzzy area, including: A first pixel value corresponding to the first fuzzy area at a first position is obtained, and a first range information corresponding to the first initial clear area is determined according to the first position and the adjacent length and the adjacent width; A second pixel value corresponding to the first initial clear area at the first range information is obtained; A first loss function between the first initial clear area and the first fuzzy area is constructed according to the first pixel value and the second pixel value; A first derivative function is obtained by performing derivative processing on the first loss function, and a second derivative function is obtained by performing Taylor expansion processing on the first derivative function; The first mapping difference value of the first fuzzy area when the features in the first fuzzy area correspond to the first initial clear area is obtained by performing displacement assumption processing on the second derivative function; The first mapping difference value is obtained according to the following formula: wherein, denotes the first mapping difference value in the first initial clear area corresponding to the feature when the horizontal direction position is x and the vertical direction position is y in the first blur area, w denotes the adjacent length, and h denotes the adjacent width, denotes the value corresponding to the second derivative function when the horizontal direction position corresponding to the first blur area is i, denotes the value corresponding to the second derivative function when the vertical direction position corresponding to the first blur area is j, denotes the first pixel value corresponding to the horizontal direction position i and the vertical direction position j in the first blur area, denotes the second pixel value corresponding to the horizontal direction position i and the vertical direction position j in the first initial clear area.

3. The method of claim 1, wherein, The target feature information corresponding to the work surface is obtained by fusing the first feature information and the second feature information, including: A target similarity value is obtained by performing similarity calculation on the first feature information and the second feature information; A target common area between the first image information and the second image information is obtained according to the target similarity value; An initial fusion image is obtained by performing image fusion on the first image information and the second image information according to the target common area; A deformation detection result is obtained by performing target deformation detection on the initial fusion image, and a target fusion image is obtained by correcting the initial fusion image according to the deformation detection result; The target feature information corresponding to the work surface is obtained by performing feature extraction on the target fusion image.

4. The method of claim 3, wherein, The deformation detection result is obtained by performing target deformation detection on the initial fusion image, including: A first sub-image corresponding to the target common area is obtained from the first image information, and a second sub-image corresponding to the target common area is obtained from the second image information; A third sub-image corresponding to the target common area after fusion is obtained from the target fusion image; calculate a first standard deviation corresponding to the first sub-image, a second standard deviation corresponding to the second sub-image, and a third standard deviation corresponding to the third sub-image; calculate a first mean value corresponding to the first sub-image, a second mean value corresponding to the second sub-image, and a third mean value corresponding to the third sub-image; obtain a first correlation value between the first sub-image and the third sub-image according to the first standard deviation, the third standard deviation, the first mean value, and the third mean value in combination with the first sub-image and the third sub-image; obtain a second correlation value between the second sub-image and the third sub-image according to the second standard deviation, the third standard deviation, the second mean value, and the third mean value in combination with the second sub-image and the third sub-image; obtain a target fusion value corresponding to the initial fusion image by fusing the first correlation value and the second correlation value; obtain the deformation detection result by performing target deformation detection on the initial fusion image according to the target fusion value.

5. The method of claim 3, wherein, The three-dimensional reconstruction according to the target feature information and the target point cloud data to obtain an initial real scene three-dimensional model, comprising: obtain target edge information corresponding to a target object in the work surface according to the target feature information; obtain target normal information and target curvature information corresponding to the target object in the work surface according to the target point cloud data; determine a target mapping relationship between the target point cloud data and the target fusion image according to the target edge information, the target normal information, and the target curvature information; calculate distance information between the target point cloud data and the target edge information according to the target mapping relationship; obtain associated point cloud data corresponding to the target edge information from the target point cloud data according to the distance information; obtain the initial real scene three-dimensional model corresponding to the work surface by three-dimensional reconstruction according to the associated point cloud data and the target edge information.

6. The method of claim 1, wherein, The dynamic detection of the work surface according to the fifth image information, the sixth image information, the seventh image information, and the eighth image information to obtain a target detection result, comprising: obtain first target information by performing target recognition on the first image information, and obtain fifth target information by performing target recognition on the fifth image information; calculate first curvature information corresponding to target segmentation of the first target information and fifth curvature information corresponding to target segmentation of the fifth target information; obtain first difference information by difference calculation according to the first curvature information and the fifth curvature information; obtain second target information by performing target recognition on the second image information, and obtain sixth target information by performing target recognition on the sixth image information; calculate second curvature information corresponding to target segmentation of the second target information and sixth curvature information corresponding to target segmentation of the sixth target information; obtain second difference information by difference calculation according to the second curvature information and the sixth curvature information; target information of the third image information and seventh target information obtained by target recognition on the seventh image information; third curvature information corresponding to target segmentation of the third target information and seventh curvature information corresponding to target segmentation of the seventh target information are calculated; third difference information is obtained by difference calculation according to the third curvature information and the seventh curvature information; fourth target information obtained by target recognition on the fourth image information and eighth target information obtained by target recognition on the eighth image information are obtained; fourth curvature information corresponding to target segmentation of the fourth target information and eighth curvature information corresponding to target segmentation of the eighth target information are calculated; fourth difference information is obtained by difference calculation according to the fourth curvature information and the eighth curvature information; The target detection result is obtained by dynamically detecting the work surface by fusing the first difference information, the second difference information, the third difference information and the fourth difference information.

7. A dynamic updating system of a real scene three-dimensional model based on a tower crane operation surface, characterized in that, It comprises: a data acquisition module, configured to obtain first image information corresponding to a first focal point at a first angle and second image information corresponding to a second focal point at the first angle of a work surface, and obtain third image information corresponding to a third focal point at a second angle and fourth image information corresponding to a fourth focal point at the second angle of the work surface, and collect target point cloud data of the work surface by using a laser radar; The first feature extraction module is configured to perform feature extraction on the first image information and the second image information according to the first focus and the second focus to obtain first feature information corresponding to the first angle, including: performing sharpness analysis on the first image information to obtain a first blur area corresponding to the first image information; and performing sharpness analysis on the second image information to obtain a second blur area corresponding to the second image information; determining an adjacent range between the same pixel points between the first image information and the second image information, the adjacent range including an adjacent length and an adjacent width; determining a first initial clear area corresponding to the first blur area in the second image information according to the first blur area and the adjacent length and the adjacent width; determining a second initial clear area corresponding to the second blur area in the first image information according to the second blur area and the adjacent length and the adjacent width; performing data alignment on the first blur area and the first initial clear area to obtain a first target clear area corresponding to the first blur area in the second image information, and performing data alignment on the second blur area and the second initial clear area to obtain a second target clear area corresponding to the second blur area in the first image information, including: performing loss calculation on the first initial clear area and the first blur area by using the adjacent length and the adjacent width to obtain a first mapping difference value when features in the first blur area correspond to the first initial clear area; performing loss calculation on the second initial clear area and the second blur area by using the adjacent length and the adjacent width to obtain a second mapping difference value when features in the second blur area correspond to the second initial clear area; determining the first target clear area corresponding to the first blur area in the second image information according to the first mapping difference value and the first initial clear area; determining the second target clear area corresponding to the second blur area in the first image information according to the second mapping difference value and the second initial clear area; performing image fusion on the first target clear area and the first image information to obtain first fusion information; performing image fusion on the second target clear area and the second image information to obtain second fusion information; and determining a first fusion image corresponding to the working surface at the first angle according to the first fusion information and the second fusion information; performing feature extraction on the first fusion image to obtain first feature information corresponding to the working surface at the first angle; The second feature extraction module is configured to perform feature extraction on the third image information and the fourth image information according to the third focus and the fourth focus to obtain second feature information corresponding to the second angle; The information fusion module is configured to perform information fusion on the first feature information and the second feature information to obtain target feature information corresponding to the working surface; The three-dimensional reconstruction module is configured to perform three-dimensional reconstruction on the target feature information and the target point cloud data to obtain an initial real scene three-dimensional model. a data updating module, configured to update the first image information, the second image information, the third image information and the fourth image information in real time to obtain fifth image information corresponding to the first image information, sixth image information corresponding to the second image information, seventh image information corresponding to the third image information and eighth image information corresponding to the fourth image information; a dynamic detection module, configured to perform dynamic detection on the work surface according to the fifth image information, the sixth image information, the seventh image information and the eighth image information to obtain a target detection result; a model updating module, configured to perform dynamic updating on the initial real-scene three-dimensional model according to the target detection result to obtain a target real-scene three-dimensional model corresponding to the work surface.

8. A terminal device, comprising: The terminal device comprises a processor and a memory; the memory is configured to store a computer program; the processor is configured to execute the computer program and implement the method for dynamically updating a real-scene three-dimensional model based on a tower crane work surface according to any one of claims 1 to 6 when the computer program is executed.

Images