Tower crane panoramic digital assembly system and method

Abstract

The present invention discloses a panoramic digital assembly system and method for a tower crane. When generating a panoramic image of a tower crane, the present invention first generates a splicing energy map for measuring the image difference between a projection image and a reference image in a corresponding overlapping area; then, based on the splicing energy map, an optimal splicing line is found that minimizes the image difference between the projection image and the reference image in the corresponding overlapping area; finally, each projection image and the reference image are spliced ​​along the optimal splicing line, thereby obtaining a panoramic exterior image of the tower crane; in this way, the present invention minimizes the image difference between the reference image and the projection image in the corresponding overlapping area by finding the optimal splicing line between the images. Based on this, splicing ghosting and blurring caused by factors such as texture differences between different images can be reduced or even avoided, thereby improving the quality of the generated panoramic exterior image of the tower crane, thereby helping operators to accurately evaluate the hoisting environment.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent tower cranes, and in particular to a panoramic digital assembly system and method for tower cranes. Background Art

[0002] The remote intelligent control system for tower cranes is an important achievement of technological progress in the construction industry in recent years. It realizes remote and centralized control of tower crane groups through a digital cockpit. Relying on 5G networks and AI visual recognition technology, it can respond to and coordinate data such as the operating trajectories and lifting loads of multiple towers in milliseconds. The system significantly improves construction safety and efficiency.

[0003] Currently, when performing remote intelligent control of tower cranes, it is necessary to generate a panoramic video of the tower crane to help operators fully understand the lifting environment. Among them, the panoramic video of the tower crane is a comprehensive view of the environment around the tower crane generated by panoramic stitching technology. Specifically, it uses multiple image acquisition devices to obtain video images within a 360-degree range around the tower crane, and uses a video stitching processing device to synthesize these images into a seamless panoramic view. This technology can help operators understand the surrounding environment more comprehensively, thereby improving the safety and accuracy of operations.

[0004] In practical applications, after mapping different images to the same coordinate system, the pixel values ​​in the overlapping areas of the images are usually directly taken as the linear weighted values ​​between the corresponding pixels of the original images; however, due to the differences between different images, if linear weighting is directly performed, it is easy to cause the stitched panoramic image to have ghosting or blurring in the overlapping areas; based on this, the quality of the generated panoramic image will be reduced, which may affect the operator's accurate assessment of the surrounding hoisting environment; therefore, based on the aforementioned shortcomings, how to provide a tower crane panoramic digital assembly system that can improve the quality of panoramic images has become an urgent problem to be solved. Summary of the Invention

[0005] The present invention aims to solve the problem of poor quality in generating panoramic video images of tower cranes. The purpose is to provide a system and method for digitizing the panoramic view of tower cranes, which solves the problem that ghosting or blurring will occur in overlapping areas when traditional technologies use linear weighted values ​​for image stitching, resulting in poor quality of panoramic images.

[0006] The present invention is achieved through the following technical solutions:

[0007] In a first aspect, a panoramic digital assembly system for a tower crane is provided, comprising:

[0008] Image acquisition equipment, used to collect monitoring images of the tower crane from different angles and transmit each monitoring image to the tower crane remote control platform;

[0009] The tower crane remote control platform is used to determine a reference image from all monitored external scene images and convert each target image into a coordinate system corresponding to the reference image to obtain a plurality of projected images, wherein each target image is a monitored external scene image excluding the reference image from all monitored external scene images;

[0010] The tower crane remote control platform is used to determine the stitching overlap area between each projection image and the reference image, and generate a stitching energy map for each stitching overlap area, wherein the stitching energy map is used to measure the image difference between the projection image and the reference image at the corresponding stitching overlap area;

[0011] The tower crane remote control platform is used to determine, from each overlapping stitching area, an optimal stitching line between each projection image and a reference image based on a stitching energy map of each overlapping stitching area, wherein the optimal stitching line is used to minimize the image difference between the projection image and the reference image in the corresponding overlapping stitching area;

[0012] The tower crane remote control platform is also used to use the best stitching line between each projection image and the reference image to stitch the reference image with each projection image, so as to obtain a panoramic exterior image of the tower crane after the stitching process.

[0013] Based on the above-disclosed content, the present invention converts each monitored external scene image into the coordinate system corresponding to the reference image to obtain a number of projection images. The stitching overlap area between each projection image and the reference image is first determined; then, a stitching energy map of each stitching overlap area is generated; wherein the stitching energy map is used to measure the image difference between the projection image and the reference image at the corresponding stitching overlap area; in this way, the present method can use the stitching energy map of each stitching overlap area to determine the optimal stitching line from each stitching overlap area that minimizes the image difference between each projection image and the reference image at the corresponding overlap area; finally, the projection image and the reference image can be stitched according to the optimal stitching line between each projection image and the reference image, thereby obtaining a panoramic external scene image of the tower crane.

[0014] Through the above design, when generating a panoramic image of a tower crane, the present invention first generates a stitching energy map for measuring the image difference between the projection image and the reference image in the corresponding overlapping area; then, based on the stitching energy map, the optimal stitching line that minimizes the image difference between the projection image and the reference image in the corresponding overlapping area is found; finally, the projection images and the reference image can be stitched along the optimal stitching line to obtain a panoramic external view image of the tower crane; in this way, the present invention minimizes the image difference between the reference image and the projection image in the corresponding overlapping area by finding the optimal stitching line between the images. Based on this, the stitching ghosting and blurring caused by factors such as texture differences between different images can be reduced or even avoided, thereby improving the quality of the generated panoramic external view image of the tower crane, thereby helping operators to accurately evaluate the hoisting environment. Therefore, the present invention is very suitable for large-scale application and promotion.

[0015] In one possible design, for any projected image, the tower crane remote control platform is configured to generate a first brightness map of the projected image and a second brightness map of the reference image, and based on the first brightness map and the second brightness map, generate a brightness difference map between the projected image and the reference image at a corresponding spliced ​​overlapping area;

[0016] The tower crane remote control platform is used to perform a convolution operation on the brightness difference map using a gradient operator to obtain a texture difference map between any one of the projection images and the reference image at a corresponding splicing overlap area;

[0017] The tower crane remote control platform is used to calculate the splicing weight of each pixel in the texture difference map, wherein the splicing weight of any pixel is calculated based on the brightness difference and texture difference of several neighboring pixels of the pixel;

[0018] The tower crane remote control platform is further used to generate a splicing energy map of the splicing overlap area between any one of the projection images and the reference image based on the splicing weight of each pixel point and the texture difference map.

[0019] In one possible design, for the jth row in the target stitching overlap area, the tower crane remote control platform is configured to determine the optimal stitching path point for each pixel point in the jth row from each pixel point in the j-1th row, where the initial value of j is 2, and the target stitching overlap area is the stitching overlap area between any projection image and the reference image;

[0020] The tower crane remote control platform is used to sequentially connect each pixel point in the j-th row, the optimal splicing path point corresponding to each pixel point, and the splicing line segment connected to each optimal splicing path point, so as to obtain an extended splicing line segment of each pixel point in the j-th row after sequential connection;

[0021] The tower crane remote control platform is used to obtain the splicing energy of each extended splicing line segment based on the splicing energy map corresponding to the target splicing overlap area;

[0022] The tower crane remote control platform is used to increment j by 1 and re-determine the optimal splicing path points for each pixel point in the j-th row from each pixel point in the j-1-th row until j equals J, and then use the extended splicing line segment of each pixel point in the j-th row as the actual splicing line, where J is the total number of rows in the target splicing overlap area;

[0023] The tower crane remote control platform is further used to screen out an actual splicing line with the minimum splicing energy from multiple actual splicing lines to serve as the optimal splicing line between any one of the projection images and the reference image.

[0024] In a possible design, for the qth pixel point in the jth row, the tower crane remote control platform is used to calculate the path node energy values ​​of all pixels in the j-1th row based on the qth pixel point;

[0025] The tower crane remote control platform is used to select the pixel point corresponding to the minimum path node energy value from each pixel point in the j-1th row to serve as the optimal splicing path point for the qth pixel point;

[0026] The tower crane remote control platform is also used to add 1 to q and recalculate the path node energy values ​​of all pixel points in the j-1th row based on the qth pixel point until q is equal to Q, thereby obtaining the optimal splicing path point of each pixel point in the jth row, where the initial value of q is 1 and Q is the total number of pixel points in the jth row.

[0027] In one possible design, for the t-th pixel in the j-1-th row, the tower crane remote control platform is used to filter out all pixels between the t-th pixel and the target pixel, where the target pixel is the pixel in the u-th column in the j-1-th row, and u is the column number where the q-th pixel is located;

[0028] The tower crane remote control platform is configured to obtain a splicing energy of an extended splicing segment corresponding to a t-th pixel point, wherein when the t-th pixel point is a pixel point in the first row within the target splicing overlap area, the splicing energy of the extended splicing segment corresponding to the t-th pixel point is a splicing energy value of the t-th pixel point, and the splicing energy value of the t-th pixel point is obtained using a splicing energy map corresponding to the target splicing overlap area;

[0029] The tower crane remote control platform is used to determine the splicing energy value of each pixel point between the t-th pixel point and the target pixel point based on the splicing energy map corresponding to the target splicing overlapping area;

[0030] The tower crane remote control platform is also used to sum the splicing energy values ​​of each pixel point between the t-th pixel point and the target pixel point, and add the sum result to the splicing energy of the extended splicing segment corresponding to the t-th pixel point to obtain the path node energy value of the t-th pixel point, where t is a positive integer.

[0031] In one possible design, the tower crane remote control platform is used to determine feature points in the reference image and each target image, and perform feature matching processing on the feature points in the reference image and each target image to obtain a set of feature point pairs between each target image and the reference image;

[0032] The tower crane remote control platform is used to perform feature point pair screening processing on each feature point pair set to obtain a plurality of feature point pair subsets corresponding to each feature point pair set, wherein the feature scales of each feature point pair subset corresponding to any feature point pair set are different;

[0033] The tower crane remote control platform is used to determine the projection transformation matrix between each target image and different areas in the reference image based on a plurality of feature point pair subsets corresponding to each feature point pair set;

[0034] The tower crane remote control platform is further used to project each target image into a coordinate system corresponding to the reference image using a projection transformation matrix between each target image and different regions in the reference image to obtain a plurality of projection images.

[0035] In a possible design, for any set of feature point pairs, the tower crane remote control platform is used to use the set of feature point pairs as a set to be screened, and randomly select two sets of feature point pairs from the set to be screened as designated feature point pairs;

[0036] The tower crane remote control platform is used to calculate the initial projection matrix using the specified feature point pairs;

[0037] The tower crane remote control platform is used to project each target feature point pair into the reference image according to the initial projection matrix, and calculate the projection error of each target feature point pair, wherein each target feature point pair is each feature point pair after removing the specified feature point pair from the set to be screened;

[0038] The tower crane remote control platform is used to screen out target feature point pairs whose projection errors are less than or equal to the error threshold from each target feature point pair, and to count the total number of the screened target feature point pairs;

[0039] The tower crane remote control platform is used to determine whether the total number of the screened target feature point pairs is greater than or equal to the minimum number of feature pairs, and when it is determined that the total number is greater than the minimum number of feature pairs, use the screened target feature point pairs to form a feature point pair subset of any feature point pair set;

[0040] The tower crane remote control platform is used to delete the selected target feature point pairs from the set to be screened to obtain a new screening set;

[0041] The tower crane remote control platform is also used to update the set to be screened to the new screening set, and randomly select two groups of feature points from the set to be screened again until the total number of target feature point pairs screened out is less than the minimum number of feature pairs, thereby obtaining several feature point pair subsets corresponding to any feature point pair set.

[0042] In one possible design, the tower crane remote control platform is used to perform gridding processing on the reference image to divide the reference image into a plurality of grid areas;

[0043] For any target image, the tower crane remote control platform is used to calculate the projection weight of each grid area on each target feature point pair subset based on a plurality of target feature point pair subsets, wherein the plurality of target feature point pair subsets are each feature point pair subset corresponding to the feature point pair set between the any target image and the reference image;

[0044] The tower crane remote control platform is further used to calculate the projection transformation matrix between any target image and different areas in the reference image based on the projection weights of each grid area on each target feature point pair subset and a plurality of target feature point pair subsets.

[0045] In one possible design, for any grid area, the tower crane remote control platform is used to screen out designated feature points from each target feature point pair subset, wherein the designated feature point in any target feature point pair subset is the feature point in any target feature point pair subset that is closest to the center point of any grid area;

[0046] The tower crane remote control platform is used to calculate the projection weight of each target feature point subset of the any grid area according to each designated feature point and the center point of the any grid area and according to the following formula (1);

[0047] (1)

[0048] In the above formula (1), Indicates that any grid area has The projection weights of the target feature points to the subset, represents the coordinate vector of the center point of any grid area, Indicates the The coordinate vector of the specified feature point in the target feature point subset, represents the proportionality constant, represents the total number of target feature point pair subsets, Represents the norm operation.

[0049] In a second aspect, a method for digitizing and assembling a tower crane in a panoramic manner is provided, comprising:

[0050] The image acquisition device collects monitoring images of the exterior scene from different angles outside the tower crane and transmits each monitoring image to the tower crane remote control platform;

[0051] The tower crane remote control platform determines a reference image from all monitored external scene images, and converts each target image into a coordinate system corresponding to the reference image to obtain a plurality of projection images, wherein each target image is a monitored external scene image excluding the reference image from all monitored external scene images;

[0052] The tower crane remote control platform determines the stitching overlap area between each projection image and the reference image, and generates a stitching energy map for each stitching overlap area, wherein the stitching energy map is used to measure the image difference between the projection image and the reference image at the corresponding stitching overlap area;

[0053] The tower crane remote control platform determines the optimal stitching line between each projection image and the reference image in each stitching overlap area based on the stitching energy map of each stitching overlap area, wherein the optimal stitching line is used to minimize the image difference between the projection image and the reference image in the corresponding stitching overlap area;

[0054] The tower crane remote control platform uses the best stitching line between each projection image and the reference image to stitch the reference image with each projection image, so as to obtain a panoramic exterior image of the tower crane after the stitching process.

[0055] In the third aspect, a panoramic digital assembly device for a tower crane is provided. Taking the device as an electronic device as an example, the device includes a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the panoramic digital assembly method for a tower crane as described in the second aspect or any possible design of the second aspect.

[0056] In a fourth aspect, a storage medium is provided, on which instructions are stored. When the instructions are run on a computer, the method for digitizing the panoramic assembly of a tower crane as described in the second aspect or any possible design of the second aspect is executed.

[0057] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to execute the tower crane panoramic digital assembly method as described in the second aspect or any possible design of the second aspect.

[0058] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0059] (1) The present invention minimizes the image difference between the reference image and the projected image in the corresponding overlapping area by finding the optimal stitching line between the images. Based on this, the stitching ghosting and blurring caused by factors such as texture differences between different images can be reduced or even avoided. As a result, the quality of the generated panoramic exterior image of the tower crane can be improved, thereby helping operators to accurately evaluate the hoisting environment. Therefore, the present invention is very suitable for large-scale application and promotion.

[0060] (2) The present invention also provides a multi-feature scale feature point pair division method, which is used to divide a feature point pair set into feature point pair subsets of different scales, and thereby generate projection transformation matrices of different regions in the projection image and the reference image; then, the projection image can be projected into the reference image according to the projection transformation matrices of different regions; based on this, the present invention uses a multi-scale projection transformation matrix to perform image projection, which can avoid the problem of image misalignment caused by local registration failure compared to the traditional single projection matrix; therefore, the image registration effect can be improved, thereby ensuring the image stitching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the examples. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort. In the drawings:

[0062] Figure 1 A schematic diagram of the architecture of a panoramic digital assembly system for tower cranes provided by an embodiment of the present invention;

[0063] Figure 2 A schematic flow chart of the steps of a panoramic digital assembly method for a tower crane provided by an embodiment of the present invention;

[0064] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the following examples and accompanying drawings. The exemplary embodiments of the present invention and their descriptions are intended only to explain the present invention and are not intended to limit the present invention. It should be understood that although the terms "first," "second," and so on may be used herein to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments of the present invention. Example

[0066] See also Figure 1 As shown, the tower crane panoramic digital assembly system provided in this embodiment may include, but is not limited to: an image acquisition device and a tower crane remote control platform, wherein the image acquisition device may be, but is not limited to, cameras installed at different parts of the tower crane, and each camera is connected to the tower crane remote control platform via a network-managed gigabit switch, that is, the image acquisition device is used to acquire monitoring external scene images from different angles outside the tower crane, and transmit each monitoring external scene image to the tower crane remote control platform; in this way, the tower crane remote control platform can generate a panoramic external scene image of the tower crane based on the monitoring external scene image transmitted by the image acquisition device.

[0067] In specific applications, the tower crane remote control platform is used to determine a reference image from all monitored external scene images, and convert each target image into a coordinate system corresponding to the reference image to obtain a number of projection images; in this embodiment, the reference image can select an external scene monitoring image of the monitoring tower crane center from the monitored external scene images, or can select a monitored external scene image with the largest viewing angle and coverage as the reference image; in this way, the remaining monitored external scene images are used as target images, that is, each target image is a monitored external scene image excluding the reference image from all monitored external scene images.

[0068] Converting each target image to the coordinate system corresponding to the reference image means converting each target image to the same coordinate system for subsequent image stitching. The process of the tower crane remote control platform performing coordinate conversion on each target image is as follows:

[0069] The tower crane remote control platform is used to determine the feature points in the reference image and each target image, and perform feature matching processing on the feature points in the reference image and each target image to obtain a feature point pair set between each target image and the reference image; wherein, the extraction and matching of feature points are the basis for image registration, and the accuracy of the extraction will directly affect the stitching of subsequent images; however, the current ORB algorithm introduces the FAST feature direction to make it rotation invariant, and extracts feature points based on this. However, since the FAST feature points do not have scale-invariant information, the ORB operator does not have scale invariance. Therefore, when the image scale changes (such as image zooming or distortion caused by different viewing angles), the extracted feature points will also change, which will lead to inaccurate feature matching, and thus affect the image stitching effect; based on this, this embodiment provides a feature point extraction method in a multi-scale space to accurately extract feature points.

[0070] Among them, the specific process of feature point extraction of the tower crane remote control platform is as follows:

[0071] Taking a reference image as an example, the tower crane remote control platform is used to perform multi-scale Gaussian convolution processing on the reference image to obtain a number of Gaussian convolution images. In a specific implementation, the Gaussian function is a kernel smoothing function of a unique scale space. Therefore, in order to extract stable feature points, this embodiment constructs an image pyramid (i.e., scale space) of different scales by performing multi-scale Gaussian convolution on the reference image, and extracts feature points within the constructed scale space.

[0072] Furthermore, any Gaussian convolution image can be expressed as:

[0073] (3)

[0074] In formula (3), represents any Gaussian convolution image, represents the convolution scale, represents the reference image, represents the Gaussian function, Represents the horizontal and vertical coordinates of each pixel in the reference image.

[0075] After completing the multi-scale Gaussian convolution processing of the reference image, the tower crane remote control platform is used to calculate the image difference between the Gaussian convolution images of adjacent scales in several Gaussian convolution images to obtain several scale space images; then, it is used to determine the extreme pixel points in each scale space image, so as to use each extreme pixel point to form a set of candidate feature points.

[0076] In specific implementation, the extreme pixel points in each scale space image are local maximum points or local minimum points. For example, for any pixel point in the scale space, the neighborhood pixel points of the arbitrary pixel point are obtained, and then the pixel values ​​of the arbitrary pixel point are compared with the corresponding neighborhood pixel points. If its pixel value is greater than the pixel values ​​of all neighborhood pixel points, it is a local maximum point, and if they are all less than the pixel values ​​of all neighborhood pixel points, it is a local minimum point. In this way, based on the above method, the extreme pixel points in each scale space image can be determined.

[0077] At the same time, in order to improve the accuracy of the extracted feature points, this embodiment also provides a process for removing invalid feature points, that is, the tower crane remote control platform is also used to remove invalid feature points in the candidate feature point set, so as to obtain feature points in the reference image after removing the invalid feature points.

[0078] In specific applications, this embodiment removes invalid feature points from the candidate feature point set based on the contrast and edge features of extreme pixel points; wherein, its specific implementation process is: the tower crane remote control platform is used to calculate the local contrast of each candidate feature point, and treat the candidate feature points with local contrast less than the contrast threshold as invalid feature points, and delete them from the candidate feature point set to obtain a preselected feature point set; then, the tower crane remote control platform is also used to filter out non-edge feature points from the preselected feature point set, and use the pixel points corresponding to the filtered non-edge feature points in the reference image as feature points in the reference image.

[0079] In a specific implementation, the essence of this embodiment is to obtain the Hessian matrix of each candidate feature point to calculate the contrast and identify the edge feature points; wherein, taking any candidate feature point as an example for explanation, it can be but not limited to obtaining the second-order partial derivative of any candidate feature point in the x direction, the second-order partial derivative in the y direction, the mixed partial derivative of x and y, and the mixed partial derivative of y and x, wherein the Hessian matrix is ​​a 2×2 matrix, the first row of which is the second-order partial derivative of any candidate feature point in the x direction and the mixed partial derivative of x and y, and the second row is the mixed partial derivative of any candidate feature point with respect to y and x and the second-order partial derivative in the y direction.

[0080] In this way, after obtaining the Hessian matrix of any candidate feature point, the contrast can be calculated based on this, that is: the tower crane remote control platform is used to perform determinant calculation processing on the Hessian matrix to obtain the local contrast of any candidate feature point after the determinant calculation processing; wherein, if the local contrast of any candidate feature point is less than 0.03, it is regarded as an invalid feature point; based on this, through the above-mentioned method, the invalid feature points can be removed from the candidate feature points, thereby obtaining a pre-selected feature point set.

[0081] Among them, the reason for obtaining the contrast through the Hessian matrix is ​​that the Hessian matrix is ​​related to the curvature of the function at the extreme point. The curvature refers to the degree of curvature of the surface at a certain point. In image processing, contrast refers to the brightness difference between different areas in the image. The greater the curvature, the stronger the contrast. The determinant of the curvature matrix is ​​proportional to the contrast of the image. The greater the contrast, the smaller the determinant of the curvature matrix. Therefore, the determinant of the Hessian matrix is ​​also smaller. Based on this, the contrast of each extreme pixel point can be measured by calculating the determinant of the Hessian matrix. Furthermore, extreme points with low contrast are usually caused by image noise or illumination changes. These feature points are unstable and easily disturbed, which is not conducive to subsequent feature matching and image stitching. In this way, extreme pixel points with local contrast less than 0.03 can be regarded as invalid feature points.

[0082] Similarly, for any preselected feature point, its edge feature recognition process is as follows: the tower crane remote control platform is used to obtain the Hessian matrix corresponding to any preselected feature point. In this embodiment, the Hessian matrix of any preselected feature point has been obtained when the local contrast calculation is performed above, so it can be used directly; after obtaining the Hessian matrix of any preselected feature point, the tower crane remote control platform is used to calculate all the eigenvalues ​​of the Hessian matrix corresponding to any preselected feature point, and screen out the maximum eigenvalue and the minimum eigenvalue from all the eigenvalues; then, the tower crane remote control platform is used to calculate the principal curvature ratio of any preselected feature point based on the maximum eigenvalue and the minimum eigenvalue; finally, the tower crane remote control platform is also used to determine whether the principal curvature ratio is less than the principal curvature threshold, and when it is determined that the principal curvature ratio is less than the principal curvature threshold, it is determined that any preselected feature point is a non-edge feature point.

[0083] Optionally, one of the calculation methods for the principal curvature ratio is provided below, as shown in the following formula (4).

[0084] (4)

[0085] In the above formula (4), represents the principal curvature ratio of any preselected feature point, represents the maximum eigenvalue and the minimum eigenvalue; at the same time, for example, the principal curvature threshold can be but is not limited to being set to 8.

[0086] As mentioned above, the Hessian matrix is ​​used to characterize the curvature of extreme pixel points. The larger the curvature, the greater the degree of curvature of the surface at that point, and the more likely that the point is an edge point. Therefore, the pre-selected feature points with a principal curvature ratio less than 8 can be retained as non-edge feature points, so that the original pixel points of the non-edge feature points in the reference image can be used as feature points of the reference image.

[0087] Furthermore, since the pixel coordinates of the image will be changed during Gaussian convolution, the coordinates of the non-edge feature points obtained at this time are not the coordinates in the original image (that is, the coordinates in the reference image). Therefore, it is necessary to find the original pixel points of the non-edge feature points in the reference image so that they can be used as the final feature points in the reference image later.

[0088] Optionally, this embodiment provides one of the methods for determining the original pixel points:

[0089] In a specific implementation, for any non-edge feature point, for example, the tower crane remote control platform can, but is not limited to, perform Taylor expansion processing on the target scale image at the any non-edge feature point to obtain the Taylor expansion corresponding to the any non-edge feature point; wherein the result of the Taylor expansion is a quadratic equation about x and y, therefore, it is set to 0, and the quadratic equation is solved, and the coordinates of the original pixel point corresponding to the any non-edge feature point can be obtained based on the solution result, thereby obtaining the original pixel point corresponding to the any non-edge feature point in the reference image.

[0090] Specifically, the Taylor expansion corresponding to any non-edge feature point is:

[0091] (5)

[0092] In formula (5), Represents the Taylor expansion of any non-edge feature point, represents the pixel value of any non-edge feature point in the target scale image (the target scale image is the scale space image where any non-edge feature point is located), , represents the position vector of any non-edge feature point, and Represents the horizontal and vertical coordinates of any non-edge feature point; Represents any non-edge feature point pair The partial derivative of Represents any non-edge feature point pair The second-order partial derivative of , T represents the transpose operation.

[0093] Based on this, the partial derivative of formula (5) is obtained and set to 0, then a quadratic equation about x and y can be obtained. Based on this, the sub-pixel position of any extreme pixel point in the target scale space can be obtained by solving the quadratic equation. Then, the sub-pixel position can be reversely mapped to the reference image according to the convolution scale of the target scale image, thereby obtaining the original pixel point of any non-edge feature point in the reference image. Assuming that the coordinates of any non-edge feature point are (x1, y1), the convolution scale is τ1, and the scale of the reference image is τ0, then the coordinates of the corresponding original pixel point are: x2=x1 / (τ1×τ0), y2=y1 / (τ1×τ0).

[0094] Based on this, using the aforementioned method, the original pixel points of each non-edge feature point in the reference image can be determined, thereby determining the feature points of the reference image.

[0095] Therefore, by constructing a scale space of a multi-scale image to extract feature points, the extracted feature points can be made scale-invariant and thus will not be affected by the image scale; at the same time, by using contrast and edge features to remove invalid feature points, the accuracy of feature point extraction can be further improved, thereby improving the accuracy of image registration; based on this, an accurate image foundation can be provided for subsequent image stitching.

[0096] After the feature points in each target image and the reference image are extracted, feature matching processing can be performed. This embodiment discloses one feature matching process to obtain a set of feature point pairs between each target image and the reference image.

[0097] Optionally, take any target image and reference image as an example to illustrate:

[0098] For each feature point in any target image, the tower crane remote control platform is used to obtain the neighborhood pixel points of each feature point in the any target image, and then use the neighborhood pixel points of each feature point to form an image block corresponding to each feature point; then, the tower crane remote control platform is used to binarize the image block corresponding to each feature point to obtain a binarized image of each feature point; then, the tower crane remote control platform is used to generate a first feature description operator for each feature point based on the binarized image of each feature point (that is, the binarized values ​​of each pixel point in the binarized image are combined into a vector, which serves as the first feature description operator of the corresponding feature point).

[0099] Similarly, the tower crane remote control platform is used to generate a second feature description operator for each feature point in the reference image; then, the tower crane remote control platform is used to perform a bitwise exclusive OR operation on the second feature description operator of any feature point in the any target image and the second feature description operator of each feature point in the reference image, and sum each element in the bitwise exclusive OR operation result (after the bitwise exclusive OR operation of the two feature description operators, the result is also a binary vector) to obtain the dissimilarity between the any feature point and each feature point in the reference image; finally, the tower crane remote control platform is also used to use the feature point with the smallest dissimilarity in the reference image as the matching feature point corresponding to the any feature point, and form a feature point pair; thus, based on the aforementioned method, the matching of each feature point in the any target image and the reference image can be completed, thereby obtaining a feature point pair set between the any target image and the reference image; of course, the generation process of the remaining feature point pairs is the same, which will not be repeated here.

[0100] After obtaining the feature point pairs between each target image and the reference image, the projection mapping matrix can be calculated. In conventional techniques, when performing image registration, only the projection transformation matrix of the entire image is usually obtained. This method is susceptible to individual errors or omissions of feature points, thereby causing misalignment of the panoramic image. Based on this, this embodiment provides an improved projection method, the projection process of which is shown below.

[0101] The tower crane remote control platform is used to perform feature point pair screening processing on each feature point pair set to obtain a number of feature point pair subsets corresponding to each feature point pair set, wherein the feature scales of each feature point pair subset corresponding to any feature point pair set are different; in this embodiment, it is equivalent to performing multiple feature point extractions on each feature point pair set to divide the feature point pair set into a number of feature point pair subsets, and then obtain projection transformation matrices at different levels to maximize the retention of feature matching information of the image at different feature scales.

[0102] In the specific implementation, take any feature point pair set as an example to specifically explain the above feature point pair screening process:

[0103] For any feature point pair set, the tower crane remote control platform is first used to take the any feature point pair set as a set to be screened, and randomly select two sets of feature point pairs from the set to be screened as designated feature point pairs; then, it is used to calculate an initial projection matrix using the designated feature point pairs; in this way, after the initial projection matrix is ​​calculated, the feature points can be projected based on it, so that the feature points can be selected using the projection error, that is, the tower crane remote control platform is used to project each target feature point pair into the reference image according to the initial projection matrix, and calculate the projection error of each target feature point pair (this embodiment In the example, each target feature point pair is each feature point pair after removing the specified feature point pair from the set to be screened); then, the tower crane remote control platform is used to screen out target feature point pairs whose projection errors are less than or equal to the error threshold from each target feature point pair, and count the total number of the screened target feature point pairs; finally, the tower crane remote control platform is used to determine whether the total number of the screened target feature point pairs is greater than or equal to the minimum number of feature pairs, and when it is determined that the total number is greater than the minimum number of feature pairs, the screened target feature point pairs are used to form a feature point pair subset of any feature point pair set.

[0104] In this embodiment, the coordinates of the four feature points in the specified feature point pair can be used, and a linear equation system or singular value decomposition method can be used to calculate the homography matrix, which can be used as the initial projection matrix; of course, using feature point pairs to solve the homography matrix is ​​a common technology for image registration, and its principle will not be repeated here.

[0105] Furthermore, the projection error is calculated as follows: assuming that the target feature point pair is A1 and A2, where A2 is the feature point in the reference image, then the aforementioned initial projection matrix is ​​used to project A1 into the reference image, that is, coordinate transformation is performed to obtain the projection point A11 corresponding to A1; then, the distance between A11 and A2 is calculated, and the distance between the two is used as the projection error; in this way, the target feature point pair with a projection error less than the error threshold can be screened out as the feature point pair that meets the projection accuracy under the current projection matrix.

[0106] Therefore, based on the above operations, a feature screening of any feature point pair set can be completed; at this time, a feature point pair subset under a feature scale can be obtained; then, the target feature point pairs screened out are deleted from the original feature point pair set (i.e., the screened set), and two groups of feature point pairs are reselected to screen the feature point pairs. In this way, the feature point pairs are continuously extracted according to the above principle until the total number of the target feature point pairs screened out is less than the minimum number of feature pairs, and the screening of the feature point pairs can be ended, i.e., the tower crane remote control platform is used to delete the screened target feature point pairs from the set to be screened to obtain a new screening set; finally, the tower crane remote control platform is also used to update the set to be screened to the new screening set, and reselect two groups of feature points from the set to be screened until the total number of the target feature point pairs screened out is less than the minimum number of feature pairs, and a number of feature point pair subsets corresponding to any feature point pair set are obtained.

[0107] In this way, using the aforementioned screening method, the feature point pairs of the remaining feature point pair sets can be screened, thereby obtaining several feature point pair subsets corresponding to each feature point pair set; based on this, multiple groups of different feature point pair subsets can be extracted, and each group of feature point pair subsets corresponds to different levels of matching feature information.

[0108] After obtaining several feature point pair subsets corresponding to each feature point pair set, the projection transformation matrix between each target image and different areas in the reference image can be obtained based on this, so as to use multiple projection transformation matrices between each target image and the reference image to perform the projection of each target image, that is: the tower crane remote control platform is used to determine the projection transformation matrix between each target image and different areas in the reference image according to several feature point pair subsets corresponding to each feature point pair set, and use the projection transformation matrix between each target image and different areas in the reference image to project each target image into the coordinate system corresponding to the reference image to obtain several projection images.

[0109] Optionally, the calculation process of the projection transformation matrix between different regions in the reference image and any target image is:

[0110] The tower crane remote control platform is used to grid the reference image so as to divide the reference image into a plurality of grid areas (in this embodiment, the number of grid areas can be specifically set according to actual use and is not specifically limited here); then, the tower crane remote control platform is used to calculate the projection weight of each grid area on each target feature point pair subset based on a plurality of target feature point pair subsets (wherein, the plurality of target feature point pair subsets are the plurality of feature point pair subsets corresponding to the feature point pair set between any target image and the reference image); finally, the tower crane remote control platform is used to calculate the projection transformation matrix between any target image and different areas in the reference image based on the projection weight of each grid area on each target feature point pair subset and the plurality of target feature point pair subsets.

[0111] Furthermore, the following uses any grid area as an example to disclose one of the calculation processes of the projection weight:

[0112] Among them, for any grid area, the tower crane remote control platform is first used to screen out the designated feature points in each target feature point pair subset; in this embodiment, the designated feature point in any target feature point pair subset is the feature point in the any target feature point pair subset that is closest to the center point of the any grid area; then, it is used to calculate the projection weight of the any grid area on each target feature point pair subset according to the following formula (1) based on each designated feature point and the center point of the any grid area.

[0113] (1)

[0114] In the above formula (1), Indicates that any grid area has The projection weights of the target feature points to the subset, represents the coordinate vector of the center point of any grid area, Indicates the The coordinate vector of the specified feature point in the target feature point subset, represents the proportionality constant, represents the total number of target feature point pair subsets, represents the norm operation, where .

[0115] In this way, the coordinate vector of the specified feature point in each target feature point pair subset is substituted into the aforementioned formula (1), and the projection weight of any grid area on each target feature point pair subset can be obtained; similarly, the coordinate vector of the center point of each remaining grid area is determined, and then the coordinate vector of the center point of each remaining grid area and the coordinate vector of the feature point in each target feature point pair subset that is closest to the center point of each grid area are substituted into the aforementioned formula (1), and the projection weight of the remaining grid area on each target feature point pair subset can be calculated.

[0116] After obtaining the projection weights of each grid area in the reference image to each subset of target feature point pairs, the projection transformation matrix between any target image and different areas in the reference image can be calculated.

[0117] Here, we take any grid area as an example to illustrate the calculation process:

[0118] The tower crane remote control platform is first used to calculate the projection matrix of each target feature point pair subset based on each target feature point pair subset; then, the tower crane remote control platform is also used to calculate the projection transformation matrix between the any target image and the target area in the reference image based on the projection matrix of each target feature point pair subset and the projection weight of the any grid area on each target feature point pair subset, and according to the following formula (2); in this embodiment, the target area is the any grid area.

[0119] (2)

[0120] In the above formula (2), represents the projection transformation matrix between any target image and the target area, Indicates the The projection matrix corresponding to the subset of target feature points.

[0121] In this embodiment, for example, but not limited to, the least squares method can be used to obtain the homography matrix of each target feature point subset, thereby serving as the projection matrix of each target feature point subset; of course, the least squares method is a commonly used algorithm for obtaining the homography matrix of the feature point pair, and its principle will not be repeated here.

[0122] In this way, by dividing the reference image into multiple grids and calculating the homography matrix of each grid separately, the transformation relationship between images can be described more accurately, thereby improving the image stitching accuracy.

[0123] After obtaining the projection transformation matrix between each target image and different areas in the reference image, the target image can be divided into different areas (the divided areas are consistent with the reference image); then, the different areas of the target image are transformed using the projection transformation matrix of the corresponding area to perform coordinate transformation of the pixel points in each area of ​​the target image, thereby converting each target image to the coordinate system corresponding to the reference image to obtain several projection images.

[0124] Therefore, after the projection of each target image is completed, image stitching can be performed. In this embodiment, an image stitching method based on the optimal stitching line is provided, that is, the tower crane remote control platform is first used to determine the stitching overlap area between each projected image and the reference image (in this embodiment, the intersection between the point where the target image is projected into the reference image and the boundary of the reference image is the overlap area), and generate a stitching energy map of each stitching overlap area (the stitching energy map is used to measure the image difference between the projected image and the reference image at the corresponding stitching overlap area); then, the tower crane remote control platform is used to, based on the stitching energy map of each stitching overlap area, An optimal stitching line is determined between each projected image and the reference image. In this embodiment, the optimal stitching line is used to minimize the image difference between the projected image and the reference image in the corresponding stitching overlap area. Thus, this embodiment is equivalent to minimizing the image difference between the reference image and the projected image in the corresponding overlap area by finding the optimal stitching line between the images, thereby reducing or even avoiding stitching ghosting and blurring caused by factors such as texture differences between different images. Finally, the tower crane remote control platform is used to use the optimal stitching line between each projected image and the reference image to stitch the reference image with each projected image, so as to obtain a panoramic exterior image of the tower crane after the stitching process.

[0125] In specific applications, this embodiment constructs a splicing energy map, that is, a splicing energy function, according to the following two points. The first point is that the brightness difference between the corresponding points on the splicing line and the two original images is the smallest; the second point is that the texture structures of the corresponding points on the splicing line and the two original images are the most similar.

[0126] Therefore, this implementation is based on this to construct a spliced ​​energy image, and the process is as follows:

[0127] For any projection image, the tower crane remote control platform is used to generate a first brightness map of the any projection image and a second brightness map of the reference image, and based on the first brightness map and the second brightness map, generate a brightness difference map between the any projection image and the reference image at the corresponding spliced ​​overlapping area; in this embodiment, it is equivalent to first converting the aforementioned any projection image and the reference image into a brightness map, and then, from the brightness map of any projection image, filtering out the pixel points that overlap with the reference image after projection, and then, using the filtered pixel points as designated pixel points; finally, using the brightness value of the designated pixel point in the first brightness map, subtracting the brightness value of the pixel point in the second brightness map that overlaps with the designated pixel point, so that a brightness difference map between the two at the corresponding spliced ​​overlapping area can be obtained.

[0128] Optionally, the first brightness map of any projected image is generated according to the following formula (6).

[0129] (6)

[0130] In the above formula (6), Indicates the brightness value, Represents the red component value, green component value, and blue component value of each pixel in any projected image.

[0131] After obtaining the brightness difference map, texture difference analysis can be performed based on it, that is: the tower crane remote control platform is used to use a gradient operator to perform a convolution operation on the brightness difference map to obtain a texture difference map between any one of the projection images and the reference image in the corresponding spliced ​​overlapping area; in specific applications, for example, but not limited to, the Roberts operator can be used to perform a convolution operation on the brightness difference map, that is, two 2×2 convolution kernels are used to calculate the horizontal and vertical gradients of the image, and the gradient reflects the distribution of texture differences in the overlapping area. Based on this, a texture difference map can be obtained.

[0132] After obtaining the texture difference map, in order to associate it with the brightness difference, this embodiment proposes a weight calculation formula that combines the brightness difference and the texture difference, so as to assign a splicing weight to each pixel point in the texture difference map, that is: the tower crane remote control platform is used to calculate the splicing weight of each pixel point in the texture difference map, and the splicing weight of any pixel point is calculated based on the brightness difference and texture difference of several neighboring pixel points of the any pixel point.

[0133] Optionally, for any pixel point in the texture difference map, the tower crane remote control platform can, but is not limited to, first obtain several neighboring pixel points of the any pixel point; then, calculate the splicing weight of the any pixel point based on the brightness values ​​and texture difference values ​​of the several neighboring pixel points of the any pixel point.

[0134] For example, but not limited to, the following formula (7) can be used to calculate the splicing weight of any pixel point.

[0135] (7)

[0136] In the above formula (7), represents the splicing weight of any pixel point, represents the brightness value of the nth neighboring pixel among several neighboring pixels of any pixel, Represents the texture difference value of the nth neighborhood pixel, Respectively represent brightness weight and texture weight; in this embodiment, eight neighborhood pixels are taken as an example to calculate the splicing weight, and the brightness weight is set to 0.1 and the texture weight is set to 0.9.

[0137] In this way, after the stitching weight of each pixel in the texture difference map is calculated by the aforementioned formula (7), a stitching energy map can be obtained based on this, that is: the tower crane remote control platform is also used to generate a stitching energy map of the stitching overlap area between any one of the projection images and the reference image based on the stitching weight of each pixel and the texture difference map; in this embodiment, a weight matrix is ​​formed by using each stitching weight, and then multiplied with the texture difference image to obtain the stitching energy map.

[0138] Based on this, on the energy map, the smaller the energy value of a point, the more similar the color and texture of the two images at that point; therefore, through the path planning algorithm, by searching for a path with the smallest sum of energy values ​​on the energy map, the optimal stitching line can be obtained.

[0139] In the specific implementation, any projected image is taken as an example to illustrate the optimization process of the optimal stitching line, wherein the stitching overlap area between any projected image and the reference image is used as the target stitching overlap area; therefore, each pixel point in the first row of the target stitching overlap area can be used as the starting point of a stitching line; at this time, starting from the second row, an optimal stitching path point can be found for each row of pixels in the previous row, until the last row is found, and several actual stitching lines can be obtained; finally, the actual stitching line with the smallest stitching energy is screened out, which can be used as the optimal stitching line between any projected image and the reference image.

[0140] In specific applications, this embodiment improves the traditional path planning algorithm. Among them, when searching for path points, the traditional technology can only select from the three adjacent points directly opposite the current node in the row above. This greatly limits the direction of the splicing line, that is, the horizontal distance between the current point and its corresponding path node is relatively short. Based on this, the splicing line may pass through obstacles in the overlapping area, thereby affecting the splicing effect; therefore, this embodiment selects path nodes from all points in the row above the current point, thereby broadening the selection range of path nodes, and at the same time improving the selection basis of path nodes (based on the splicing energy value of the point, the splicing energy of the splicing line connected by the point is introduced), thereby weakening or even eliminating the aforementioned defects.

[0141] Optionally, the improved path planning process is:

[0142] For the j-th row in the target stitching overlap area, the tower crane remote control platform is used to determine the optimal stitching path point for each pixel point in the j-th row from each pixel point in the j-1-th row, where the initial value of j is 2, and the target stitching overlap area is the stitching overlap area between any projection image and the reference image.

[0143] In this embodiment, it is equivalent to starting from the second row to find the best splicing path point for each pixel point in the second row in the first row, wherein the selection process is: for the qth pixel point in the jth row, the tower crane remote control platform is used to calculate the path node energy value of all pixel points in the j-1th row based on the qth pixel point; then, the tower crane remote control platform is used to screen out the pixel point corresponding to the minimum path node energy value from each pixel point in the j-1th row as the best splicing path point for the qth pixel point; finally, the tower crane remote control platform is also used to add 1 to q and recalculate the path node energy value of all pixel points in the j-1th row based on the qth pixel point, until q is equal to Q, and the best splicing path point for each pixel point in the jth row is obtained; in specific implementation, the initial value of q is 1, and Q is the total number of each pixel point in the jth row.

[0144] Furthermore, the calculation process of the path node energy value of each pixel point in the j-1th row is:

[0145] For the t-th pixel point in the j-1-th row, the tower crane remote control platform is used to filter out all pixel points between the t-th pixel point and the target pixel point, wherein the target pixel point is the pixel point in the u-th column in the j-1-th row, and u is the column number where the q-th pixel point is located.

[0146] This embodiment is illustrated by an example. Assuming that t is 1 and the column number where the qth pixel in the jth row is located is 4, then the pixels between the first pixel and the fourth pixel in the j-1th row are screened out, that is, the second pixel and the third pixel in the j-1th row; of course, the above examples are only examples and are not regarded as limitations of this application.

[0147] After obtaining all the pixel points between the t-th pixel point and the target pixel point, it is also necessary to obtain the splicing energy of the extended splicing segment corresponding to the t-th pixel point, that is: the tower crane remote control platform is used to obtain the splicing energy of the extended splicing segment corresponding to the t-th pixel point, wherein, when the t-th pixel point is a pixel point in the first row within the target splicing overlapping area, the splicing energy of the extended splicing segment corresponding to the t-th pixel point is the splicing energy value of the t-th pixel point, and the splicing energy value of the t-th pixel point is obtained using the splicing energy map corresponding to the target splicing overlapping area.

[0148] In this embodiment, the explanation is still based on the above example. Assuming that j is 2 and t is 1, then the t-th pixel in the j-1-th row is the first pixel in the first row. At this time, it is the starting point of a splicing line. Therefore, its corresponding extended splicing line segment is itself. In this way, the splicing energy of its corresponding extended splicing line segment is its own splicing energy value, that is, the splicing energy value of the first pixel in the first row in the splicing energy map corresponding to the target splicing overlapping area.

[0149] For another example, assuming j is 3, then the t-th pixel in the j-1-th row is the first pixel in the second row (assuming it is M), and the best path point for the first pixel in the second row in the first row is the third pixel (assuming it is B). At this time, the extended splicing segment corresponding to the t-th pixel is: the splicing segment connecting the first pixel in the second row and the third pixel in the first row (that is, line segment BM), and the line segment BM is obtained when searching for the best splicing path points for each pixel in the second row. Therefore, its splicing energy is known.

[0150] In this way, after obtaining the splicing energy of the extended splicing segment corresponding to the t-th pixel point, the tower crane remote control platform is used to determine the splicing energy value of each pixel point between the t-th pixel point and the target pixel point based on the splicing energy map corresponding to the target splicing overlapping area (it can be searched according to the position of the pixel points in the corresponding splicing energy map); finally, the tower crane remote control platform is also used to sum the splicing energy values ​​of each pixel point between the t-th pixel point and the target pixel point, and add the summed result to the splicing energy of the extended splicing segment corresponding to the t-th pixel point to obtain the path node energy value of the t-th pixel point (t is a positive integer); of course, in this embodiment, if the t-th pixel point in the j-1-th row has the same column number as the q-th pixel point in the j-th row, or there are no other pixels between the t-th pixel point and the target pixel point, then the sum of the splicing energy values ​​of each pixel point between the t-th pixel point and the target pixel point is 0.

[0151] In this embodiment, the explanation is still based on the above examples. When j is 2 and t is 1, the path node energy value of the first pixel point in the first row = the extended splicing segment corresponding to the first pixel point in the first row (its own splicing energy value) + the splicing energy value of the third pixel point in the first row + the splicing energy value of the fourth pixel point in the first row; similarly, when j is 3, the path node energy value of the first pixel point in the second row = the extended splicing segment corresponding to the first pixel point in the second row (i.e., line segment BM) + the splicing energy value of the third pixel point in the second row + the splicing energy value of the fourth pixel point in the second row; of course, the above examples are only examples.

[0152] In this way, through the above operations, the path node energy value of each pixel point in the j-1th row can be calculated; then, the pixel point with the smallest path node energy value can be used as the optimal splicing path point for the qth pixel point in the jth row; based on this, in the above manner, the optimal splicing path points of the remaining pixel points in the jth row can be determined, and then, the nodes can be connected, that is: the tower crane remote control platform is used to connect each pixel point in the jth row, the optimal splicing path point corresponding to each pixel point, and the splicing line segment connected to each optimal splicing path point in sequence, so as to obtain the extended splicing line segment of each pixel point in the jth row after the sequential connection; based on this, the search of a section of the splicing line can be completed.

[0153] The following is an example to illustrate: Assume that the best splicing path point of the fourth pixel in the second row (assuming it is V) in the first row is: the second pixel in the first row (assuming it is F). At this time, the fourth pixel in the second row is connected to the second pixel in the first row. At the same time, since the second pixel in the first row is the starting point, it does not have a connected splicing segment. Therefore, the extended splicing segment of the fourth pixel in the second row is: segment VF. For another example, assuming J is 3, the fourth pixel in the third row ( Assuming it is S), the best stitching path point in the second row is: the third pixel point in the second row (assuming it is E), among which the best stitching path point corresponding to E in the previous row is: the fifth pixel point in the first row (assuming it is V2), that is, the stitching segment connected by the best stitching path point is EV2, then the extended stitching segment of the fourth pixel point in the third is: segment SE + segment EV2; of course, when j is other values, the generation process of the extended stitching segments of each pixel point in its row is the same as the above example, and will not be repeated here.

[0154] After obtaining the extended splicing line segments of each pixel point in the j-th row, the splicing energy of each extended splicing line segment can be calculated, that is: the tower crane remote control platform is used to obtain the splicing energy of each extended splicing line segment based on the splicing energy map corresponding to the target splicing overlapping area; wherein, for the q-th pixel point in the j-th row, the splicing energy of the corresponding extended splicing line segment is calculated as: E1+E2+E3; wherein E1 is the splicing energy value of the q-th pixel point, E2 is the path node energy value of the optimal splicing path node of the q-th pixel point, and E3 is the splicing energy of the splicing line segment connected by the optimal splicing path node of the q-th pixel point.

[0155] In this way, through the above operations, the selection of the best stitching path points for each pixel point in the j-th row can be completed; then, based on this principle, the selection of the best stitching path points for the pixel points in the remaining rows can be carried out, that is: the tower crane remote control platform is used to add 1 to j and re-determine the best stitching path points for each pixel point in the j-th row from each pixel point in the j-1-th row, until j is equal to J, and the extended stitching line segment of each pixel point in the j-th row is used as the actual stitching line, where J is the total number of rows of the target stitching overlap area.

[0156] Through the above-mentioned cyclic search, when the last row in the target overlapping area is polled, the extended stitching line segment and the corresponding stitching energy of each pixel point in the last row can be obtained; at this time, the extended stitching line segment of each pixel point in the aforementioned last row can be used as the actual stitching line segment; finally, the best stitching line is selected according to the stitching energy of the actual stitching line segment, that is: the tower crane remote control platform is also used to screen out the actual stitching line with the smallest stitching energy from multiple actual stitching lines, as the best stitching line between any one of the projection images and the reference image.

[0157] Therefore, through the aforementioned path planning algorithm, the optimal stitching line between each projection image and the reference image can be determined from each stitching overlap area; finally, the optimal stitching line between each projection image and the reference image can be used to stitch the projection images, thereby obtaining a panoramic exterior image of the tower crane.

[0158] Through the above-mentioned design, the present invention broadens the selection of path nodes to all points in the previous row of the current point. At the same time, the selection basis is the path node energy value of each point in the previous row, and the path node energy value is: the splicing energy of the splicing line segment corresponding to the point, and the sum of the splicing energy values ​​of each point between the point and the pixel points in the column where the current point is located; thus, through the above-mentioned design, the selection of path nodes not only depends on the splicing energy of the pixel points in the previous row, but also integrates the splicing energy of the pixel points and the splicing energy of the splicing line of the pixel points in the previous row; based on this, any point in the previous row can become a splicing node on the splicing line, based on this, the direction of the splicing line can be widened, so that the path nodes between two adjacent rows can have a larger distance in the horizontal direction, so that the splicing line can have the ability to bypass obstacles; thereby, the flexibility of the splicing line can be increased, thereby improving the splicing effect.

[0159] Finally, the tower crane remote control platform will display the generated panoramic external image on the display screen of the ground digital cockpit, thereby providing the operator with an accurate image of the environment around the tower crane to assist the operator in lifting work.

[0160] Therefore, through the above detailed explanation of the panoramic digital assembly system of the tower crane, the present invention minimizes the image difference between the reference image and the projected image in the corresponding overlapping area by finding the optimal stitching line between the images. Based on this, the stitching ghosting and blurring caused by factors such as texture differences between different images can be reduced or even avoided. As a result, the quality of the generated panoramic exterior image of the tower crane can be improved, thereby helping operators to accurately evaluate the hoisting environment. Therefore, the present invention is very suitable for large-scale application and promotion.

[0161] like Figure 2As shown, the second aspect of this embodiment provides a method for the panoramic digital assembly of a tower crane based on the panoramic digital assembly system of the tower crane described in the first aspect of the embodiment; wherein, for example, the method can be but is not limited to running on the image acquisition device and the tower crane remote control platform side; of course, the aforementioned execution subject does not constitute a limitation on the embodiment of the present application, and accordingly, the operation steps of this method can be but are not limited to the following steps S1 to S5.

[0162] S1. The image acquisition device acquires monitoring images of the exterior of the tower crane from different angles and transmits each monitoring image to the tower crane remote control platform.

[0163] S2. The tower crane remote control platform determines a reference image from all monitored external images and converts each target image into a coordinate system corresponding to the reference image to obtain several projected images, where each target image is a monitored external image excluding the reference image from all monitored external images.

[0164] S3. The tower crane remote control platform determines the stitching overlap region between each projection image and the reference image, and generates a stitching energy map for each stitching overlap region. The stitching energy map is used to measure the image difference between the projection image and the reference image at the corresponding stitching overlap region.

[0165] S4. The tower crane remote control platform determines, based on the stitching energy maps of each stitching overlap region, an optimal stitching line between each projection image and the reference image in each stitching overlap region, wherein the optimal stitching line is used to minimize the image difference between the projection image and the reference image in the corresponding stitching overlap region.

[0166] S5. The tower crane remote control platform uses the best stitching lines between each projection image and the reference image to stitch the reference image with each projection image, so as to obtain a panoramic exterior image of the tower crane after the stitching process.

[0167] The working process, working details and technical effects of the method provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0168] like Figure 3 As shown, the third aspect of this embodiment provides a panoramic digital assembly device for a tower crane. Taking the device as an electronic device as an example, it includes: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the panoramic digital assembly method for a tower crane as described in the second aspect of the embodiment.

[0169] For example, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in first-out memory (FIFO), and / or first-in last-out memory (FILO). Specifically, the processor may include one or more processing cores, such as a quad-core processor or an octal-core processor. The processor may be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Furthermore, the processor may include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state.

[0170] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. For example, the processor may be, but is not limited to, a microprocessor of the STM32F105 series, a reduced instruction set computer (RISC) microprocessor, an X86 architecture processor, or a processor with an integrated embedded neural network processing unit (NPU). The transceiver may be, but is not limited to, a Wireless Fidelity (WIFI) wireless transceiver, a Bluetooth wireless transceiver, a General Packet Radio Service (GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0171] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0172] The fourth aspect of this embodiment provides a storage medium that stores instructions for the panoramic digital assembly method of a tower crane as described in the second aspect of the embodiment, that is, the storage medium stores instructions, and when the instructions are run on a computer, the panoramic digital assembly method of a tower crane as described in the second aspect of the embodiment is executed.

[0173] The storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0174] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the second aspect of the embodiment and will not be described in detail here.

[0175] The fifth aspect of this embodiment provides a computer program product containing instructions, which, when executed on a computer, enables the computer to execute the tower crane panoramic digital assembly method as described in the second aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0176] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A panoramic digital assembly system for tower cranes, characterized by: include: Image acquisition equipment, used to collect monitoring images of the tower crane from different angles and transmit each monitoring image to the tower crane remote control platform; The tower crane remote control platform is used to determine a reference image from all monitored external scene images and convert each target image into a coordinate system corresponding to the reference image to obtain a plurality of projected images, wherein each target image is a monitored external scene image excluding the reference image from all monitored external scene images; The tower crane remote control platform is used to determine the stitching overlap area between each projection image and the reference image, and generate a stitching energy map for each stitching overlap area, wherein the stitching energy map is used to measure the image difference between the projection image and the reference image at the corresponding stitching overlap area; The tower crane remote control platform is used to determine, from each overlapping stitching area, an optimal stitching line between each projection image and a reference image based on a stitching energy map of each overlapping stitching area, wherein the optimal stitching line is used to minimize the image difference between the projection image and the reference image in the corresponding overlapping stitching area; The tower crane remote control platform is further used to use the best stitching line between each projection image and the reference image to stitch the reference image with each projection image, so as to obtain a panoramic exterior image of the tower crane after the stitching process; The tower crane remote control platform is used to determine the feature points in the reference image and each target image, and perform feature matching processing on the feature points in the reference image and each target image to obtain a feature point pair set between each target image and the reference image; The tower crane remote control platform is used to perform feature point pair screening processing on each feature point pair set to obtain a plurality of feature point pair subsets corresponding to each feature point pair set, wherein the feature scales of each feature point pair subset corresponding to any feature point pair set are different; The tower crane remote control platform is used to determine the projection transformation matrix between each target image and different areas in the reference image based on a plurality of feature point pair subsets corresponding to each feature point pair set; The tower crane remote control platform is further used to project each target image into a coordinate system corresponding to the reference image using a projection transformation matrix between each target image and different regions in the reference image to obtain a plurality of projection images.

2. A tower crane panoramic digital assembly system according to claim 1, characterized in that: For any projected image, the tower crane remote control platform is used to generate a first brightness map of the projected image and a second brightness map of the reference image, and based on the first brightness map and the second brightness map, generate a brightness difference map between the projected image and the reference image at a corresponding spliced ​​overlapping area; The tower crane remote control platform is used to perform a convolution operation on the brightness difference map using a gradient operator to obtain a texture difference map between any one of the projection images and the reference image at a corresponding splicing overlap area; The tower crane remote control platform is used to calculate the splicing weight of each pixel in the texture difference map, wherein the splicing weight of any pixel is calculated based on the brightness difference and texture difference of several neighboring pixels of the pixel; The tower crane remote control platform is further used to generate a splicing energy map of the splicing overlap area between any one of the projection images and the reference image based on the splicing weight of each pixel point and the texture difference map.

3. A tower crane panoramic digital assembly system according to claim 1, characterized in that: For the j-th row in the target stitching overlap area, the tower crane remote control platform is used to determine the optimal stitching path point for each pixel point in the j-th row from each pixel point in the j-1-th row, where the initial value of j is 2, and the target stitching overlap area is the stitching overlap area between any projection image and the reference image; The tower crane remote control platform is used to sequentially connect each pixel point in the j-th row, the optimal splicing path point corresponding to each pixel point, and the splicing line segment connected to each optimal splicing path point, so as to obtain an extended splicing line segment of each pixel point in the j-th row after sequential connection; The tower crane remote control platform is used to obtain the splicing energy of each extended splicing line segment based on the splicing energy map corresponding to the target splicing overlap area; The tower crane remote control platform is used to increment j by 1 and re-determine the optimal splicing path points for each pixel point in the j-th row from each pixel point in the j-1-th row until j equals J, and then use the extended splicing line segment of each pixel point in the j-th row as the actual splicing line, where J is the total number of rows in the target splicing overlap area; The tower crane remote control platform is further used to screen out an actual splicing line with the minimum splicing energy from multiple actual splicing lines to serve as the optimal splicing line between any one of the projection images and the reference image.

4. A tower crane panoramic digital assembly system according to claim 3, characterized in that: For the qth pixel point in the jth row, the tower crane remote control platform is used to calculate the path node energy values ​​of all pixel points in the j-1th row based on the qth pixel point; The tower crane remote control platform is used to select the pixel point corresponding to the minimum path node energy value from each pixel point in the j-1th row to serve as the optimal splicing path point for the qth pixel point; The tower crane remote control platform is also used to add 1 to q and recalculate the path node energy values ​​of all pixel points in the j-1th row based on the qth pixel point until q is equal to Q, thereby obtaining the optimal splicing path point of each pixel point in the jth row, where the initial value of q is 1 and Q is the total number of pixel points in the jth row.

5. A tower crane panoramic digital assembly system according to claim 4, characterized in that: For the t-th pixel in the j-1-th row, the tower crane remote control platform is used to filter out all pixels between the t-th pixel and the target pixel, where the target pixel is the pixel in the u-th column in the j-1-th row, and u is the column number where the q-th pixel is located; The tower crane remote control platform is configured to obtain a splicing energy of an extended splicing segment corresponding to a t-th pixel point, wherein when the t-th pixel point is a pixel point in the first row within the target splicing overlap area, the splicing energy of the extended splicing segment corresponding to the t-th pixel point is a splicing energy value of the t-th pixel point, and the splicing energy value of the t-th pixel point is obtained using a splicing energy map corresponding to the target splicing overlap area; The tower crane remote control platform is used to determine the splicing energy value of each pixel point between the t-th pixel point and the target pixel point based on the splicing energy map corresponding to the target splicing overlapping area; The tower crane remote control platform is also used to sum the splicing energy values ​​of each pixel point between the t-th pixel point and the target pixel point, and add the sum result to the splicing energy of the extended splicing segment corresponding to the t-th pixel point to obtain the path node energy value of the t-th pixel point, where t is a positive integer.

6. The panoramic digital assembly system for tower cranes according to claim 1, characterized in that: For any feature point pair set, the tower crane remote control platform is used to use the any feature point pair set as a set to be screened, and randomly select two groups of feature point pairs from the set to be screened as designated feature point pairs; The tower crane remote control platform is used to calculate the initial projection matrix using the specified feature point pairs; The tower crane remote control platform is used to project each target feature point pair into the reference image according to the initial projection matrix, and calculate the projection error of each target feature point pair, wherein each target feature point pair is each feature point pair after removing the specified feature point pair from the set to be screened; The tower crane remote control platform is used to screen out target feature point pairs whose projection errors are less than or equal to the error threshold from each target feature point pair, and to count the total number of the screened target feature point pairs; The tower crane remote control platform is used to determine whether the total number of the screened target feature point pairs is greater than or equal to the minimum number of feature pairs, and when it is determined that the total number is greater than the minimum number of feature pairs, use the screened target feature point pairs to form a feature point pair subset of any feature point pair set; The tower crane remote control platform is used to delete the selected target feature point pairs from the set to be screened to obtain a new screening set; The tower crane remote control platform is also used to update the set to be screened to the new screening set, and randomly select two groups of feature points from the set to be screened again until the total number of target feature point pairs screened out is less than the minimum number of feature pairs, thereby obtaining several feature point pair subsets corresponding to any feature point pair set.

7. The panoramic digital assembly system for tower cranes according to claim 1, characterized in that: The tower crane remote control platform is used to perform grid processing on the reference image to divide the reference image into a plurality of grid areas; For any target image, the tower crane remote control platform is used to calculate the projection weight of each grid area on each target feature point pair subset based on a plurality of target feature point pair subsets, wherein the plurality of target feature point pair subsets are each feature point pair subset corresponding to the feature point pair set between the any target image and the reference image; The tower crane remote control platform is further used to calculate the projection transformation matrix between any target image and different areas in the reference image based on the projection weights of each grid area on each target feature point pair subset and a plurality of target feature point pair subsets.

8. A tower crane panoramic digital assembly system according to claim 7, characterized in that: For any grid area, the tower crane remote control platform is used to screen out designated feature points in each target feature point pair subset, wherein the designated feature point in any target feature point pair subset is the feature point in any target feature point pair subset that is closest to the center point of any grid area; The tower crane remote control platform is used to calculate the projection weight of each target feature point subset of the any grid area according to each designated feature point and the center point of the any grid area and according to the following formula (1); （1） In the above formula (1), Indicates that any grid area has The projection weights of the target feature points to the subset, represents the coordinate vector of the center point of any grid area, Indicates the The coordinate vector of the specified feature point in the target feature point subset, represents the proportionality constant, represents the total number of target feature point pair subsets, Represents the norm operation.

9. A panoramic digital assembly method for tower cranes, characterized in that: include: The image acquisition device collects monitoring images of the exterior scene from different angles outside the tower crane and transmits each monitoring image to the tower crane remote control platform; The tower crane remote control platform determines a reference image from all monitored external scene images, and converts each target image into a coordinate system corresponding to the reference image to obtain a plurality of projection images, wherein each target image is a monitored external scene image excluding the reference image from all monitored external scene images; The tower crane remote control platform determines the stitching overlap area between each projection image and the reference image, and generates a stitching energy map for each stitching overlap area, wherein the stitching energy map is used to measure the image difference between the projection image and the reference image at the corresponding stitching overlap area; The tower crane remote control platform determines the optimal stitching line between each projection image and the reference image in each stitching overlap area based on the stitching energy map of each stitching overlap area, wherein the optimal stitching line is used to minimize the image difference between the projection image and the reference image in the corresponding stitching overlap area; The tower crane remote control platform uses the best stitching line between each projection image and the reference image to stitch the reference image and each projection image, so as to obtain a panoramic exterior image of the tower crane after the stitching process; The tower crane remote control platform is used to determine the feature points in the reference image and each target image, and perform feature matching processing on the feature points in the reference image and each target image to obtain a feature point pair set between each target image and the reference image; The tower crane remote control platform is used to perform feature point pair screening processing on each feature point pair set to obtain a plurality of feature point pair subsets corresponding to each feature point pair set, wherein the feature scales of each feature point pair subset corresponding to any feature point pair set are different; The tower crane remote control platform is used to determine the projection transformation matrix between each target image and different areas in the reference image based on a plurality of feature point pair subsets corresponding to each feature point pair set; The tower crane remote control platform is further used to project each target image into a coordinate system corresponding to the reference image using a projection transformation matrix between each target image and different regions in the reference image to obtain a plurality of projection images.

Images