A position monitoring method and system based on heterogeneous image registration and binocular positioning

Abstract

The present invention discloses a position monitoring method and system based on heterogeneous image registration and binocular positioning, which relates to the field of position monitoring. The method includes: performing target detection and instance segmentation on the image data of the dangerous area obtained by the left and right binocular cameras to construct a mask image set; pairing and processing the mask image of the target to obtain the target pairing and correction image; introducing angle fusion scoring to screen the best feature points, performing image registration on the best feature points according to the bit difference, and performing fusion display; calculating the optimal binocular parallax to obtain the three-dimensional coordinates of the target and generate a point cloud; integrating the personnel target point cloud image with the background according to the acquired crane spatial position to form a complete real-time personnel monitoring point cloud image. By combining the binocular vision technology of infrared and visible light cameras, the YOLOv8 algorithm and three-dimensional point cloud modeling, real-time monitoring of dangerous areas in lifting operations is achieved, significantly improving operation safety and monitoring efficiency.

Description

Technical Field

[0001] The present invention relates to the field of position monitoring, and in particular to a position monitoring method and system based on heterogeneous image registration and binocular positioning. Background Art

[0002] As modern hoisting operations become increasingly complex and dangerous, traditional hoisting safety monitoring methods face challenges in terms of real-time performance, accuracy, and intelligence. Traditional safety monitoring methods rely on manual inspections and single sensor devices, resulting in slow response, low accuracy, and difficulty in fully covering the entire operating area. With the rapid development of computer vision and 3D reconstruction technologies, intelligent monitoring combining data from multiple sensors, particularly those based on binocular vision and image registration, has become an effective means of on-site safety monitoring for hoisting operations.

[0003] Many existing visual monitoring methods rely on a single visual image for target detection and location, or utilize only information from either infrared or visible light images. These methods typically only work effectively under specific lighting or viewing angle conditions and struggle to adapt to complex operating environments. To address these issues, heterogeneous image registration technology is used. This combines infrared and visible light images to form a left and right binocular camera for spatial information acquisition. This effectively improves the accuracy and real-time performance of detecting people and equipment within the lifting operation area.

[0004] Furthermore, existing methods require high precision for image registration and target positioning, requiring the use of efficient image registration and target positioning technologies. However, due to the high computational complexity and poor robustness of the binocular disparity calculation or 3D reconstruction techniques used in most traditional methods, practical applications often encounter problems such as inaccurate target recognition and imprecise target matching. Therefore, accurate target matching and precise image registration from multiple perspectives, as well as precise 3D target positioning based on real-time data, remain key challenges in current research.

[0005] Based on heterogeneous image registration and binocular positioning technology, the present invention proposes an improved safety monitoring method for hoisting operation areas. By combining the image information of infrared cameras and visible light cameras, the YOLOv8 algorithm is used for target detection and instance segmentation, a target mask image set is constructed, and image matching and correction are performed with the help of energy loss function to obtain accurate target matching images. Further, by extracting image feature points and introducing angle fusion scores, the feature points are screened, and bit difference is used for image registration, ultimately achieving high-precision three-dimensional target positioning. This method combines binocular parallax calculation and triangulation principles to construct a three-dimensional point cloud model of the lifting operation site of the crane, monitor the spatial position of personnel and equipment at the operation site in real time, improve monitoring accuracy and real-time performance, and effectively ensure the safety of the lifting operation. Summary of the Invention

[0006] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a position monitoring method and system based on heterogeneous image registration and binocular positioning to solve the above-mentioned technical problems.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a position monitoring method based on heterogeneous image registration and binocular positioning, comprising:

[0008] By setting up an infrared camera and a visible light camera to form a left and right binocular camera, image data of the dangerous area of ​​the lifting operation is obtained. The YOLOv8 algorithm is used to perform target detection and instance segmentation on the image data to construct a mask image set.

[0009] By constructing an energy loss function, the mask images of each target of the left and right cameras are paired and processed to obtain the target pairing and correction image of the left camera target mask image and the right camera target mask image;

[0010] Extract the target paired image feature points and introduce angle fusion scoring. Filter the extracted feature points to obtain the best feature points. Perform image registration on the best feature points based on bit difference. Fusion display the registered left and right camera images.

[0011] The optimal binocular disparity is calculated using the collaborative registration dual energy function. The three-dimensional coordinates of the target image in real space are calculated by combining the optimal binocular disparity with the triangulation principle to construct a three-dimensional point cloud model of the lifting operation site and the target point cloud of the personnel.

[0012] The spatial position of the crane is obtained through image matching, and the target point cloud image of the personnel is moved to the background point cloud according to the spatial position coordinates of the crane to form a complete real-time personnel monitoring point cloud image.

[0013] The present invention is further configured to use the YOLOv8 algorithm to perform target detection and instance segmentation on the acquired left camera infrared image Image_left and right camera visible light image Image_right to create a target mask contour image, obtain a mask binary image of the target after training mapping, and construct a mask image set mask_left and mask_right, wherein mask_left is the left camera mask image set, and mask_right is the right camera mask image set.

[0014] The present invention is further configured to calculate the energy loss function logic: Loss match =∑(mask_left[a] i,j -mask_right[b] i,j ), where Loss match is an energy loss function. The energy loss is calculated for each mask image of the left and right cameras. The right camera target mask image mask_right[b] paired with the left camera target mask image mask_left[a] when the energy loss value is minimized is taken to form a target pair. The paired left and right mask images are corrected using binocular image stereo rectification technology to obtain the corrected left mask image rectified_mask_left and right mask image rectified_mask_right.

[0015] The present invention is further configured to use the ORB operator to extract feature points from the target pairing to obtain candidate feature points f, and the optimal feature point screening logic is: f best =max(S match (f)), where f best is the best feature point; S match (f) is the angle fusion score of the candidate feature points, and the calculation logic is: Where N is the number of feature points; d i is the distance of the i-th matching point; d max is the maximum matching distance allowed; w i is the weight of the i-th matching point; e angle,i is the angle error, calculation logic: in, and is the direction vector of the left and right camera target mask images under the i-th candidate feature point; S localgeo,i Score local geometric consistency, calculation logic: Among them, K is the number of neighborhood points of the i-th candidate feature point; d ij is the distance between the i-th matching point and the j-th matching point; w j is the weight of the j-th neighborhood point.

[0016] The present invention is further configured to use the Brief operator to calculate the descriptor of the optimal feature point, use the Brute-Force matcher on the obtained feature point and the descriptor, set the matching condition to match based on the bit difference, and obtain the matching result, wherein the bit difference calculation logic is: Among them, D(A,B) is the bit difference, N is the number of bits of the descriptor, A i and B i are the values ​​of the two descriptors at the i-th position, and ⊕ is the exclusive OR operator.

[0017] The present invention is further configured to cut out the target image from the left camera image according to the result of instance segmentation based on the matching result, and project all target points according to the perspective transformation matrix to obtain the registered left camera target image Image2; the image fusion principle is: Image3(x,y)=αImage2(x,y)+(1-α)Image_right, where Image3 is the fused image, x and y are the horizontal coordinates and vertical coordinates of the image respectively, α is the fusion weight, and the calculation logic is: Among them, G red is the gradient amplitude of the infrared image in the target area, G rgb is the gradient amplitude of the visible light image in the target area, and ∈ is the minimum value to prevent the denominator from being zero.

[0018] The present invention is further configured as follows: the collaborative registration dual energy function calculation logic S(d) = ∑ x,y |rectified_mask_left(xd,y)-rectified_mask_right(x,y)|, where d is the pixel distance of translation and S(d) is the co-registration dual energy value; the optimal binocular disparity calculation logic is: Among them, disp is the optimal binocular disparity.

[0019] The present invention is further configured to complete target positioning based on the triangulation principle, and the target positioning calculation logic is: Among them, Location (X, Y, Z) is the target location, cx and cy are the coordinates of the center point in the pixel coordinate system, B is the actual physical distance between the optical centers of the binocular cameras, and f is the focal length of the camera; the crane carries the camera to collect on-site images to form the binocular camera fusion image Image3, and uses Colmap for multi-view stereo modeling to generate a complete 3D point cloud model of the crane lifting operation site; the non-zero pixel values ​​in the rectified left camera mask image rectified_mask_left are filled with the optimal binocular disparity to obtain a disparity image, and the binocular stereo vision principle is used to restore the disparity image to a 3D point cloud image in space. The thickness δ is given according to the empirical value, and the point cloud in the generated plane point cloud image is randomly scattered. The scattered space is within the range of δ before and after the original plane to simulate the thickness of the personnel to obtain the personnel target point cloud.

[0020] The present invention is further configured to use the image registration function of Colmap to match the currently captured image with the background image in the database, set the crane as the target, obtain the real-time spatial position Location (X, Y, Z) of the crane according to the target positioning method, and map the target point cloud image of the personnel to the background point cloud based on the spatial position of the crane, so as to realize real-time monitoring and tracking of the personnel.

[0021] The present invention also provides a position monitoring system based on heterogeneous image registration and binocular positioning, the system comprising:

[0022] Image acquisition and target detection module: This module uses an infrared camera and a visible light camera to form a left and right binocular camera to acquire image data of the hazardous area during lifting operations. The YOLOv8 algorithm is used to perform target detection and instance segmentation on the image data to construct a mask image set.

[0023] Target pairing and correction module: This module pairs and processes the mask images of each target in the left and right cameras by constructing an energy loss function to obtain the target pairing and correction images of the left camera target mask image and the right camera target mask image;

[0024] Image registration and fusion module: This module extracts the image feature points of the target pair and introduces angle fusion scoring. The extracted feature points are screened to obtain the best feature points. The best feature points are then registered based on the bit difference, and the registered left and right camera images are fused and displayed.

[0025] 3D space reconstruction and point cloud generation module: This module extracts target paired image feature points and introduces angle fusion scoring. The extracted feature points are screened to obtain the best feature points. The best feature points are then registered based on bit difference, and the registered left and right camera images are fused and displayed.

[0026] Background integration and real-time monitoring module: The spatial position of the crane is obtained through image matching, and the target point cloud image of the personnel is moved to the background point cloud according to the spatial position coordinates of the crane to form a complete real-time personnel monitoring point cloud image.

[0027] The present invention provides a position monitoring method and system based on heterogeneous image registration and binocular positioning. The method acquires image data of a dangerous area of ​​a lifting operation by arranging left and right binocular cameras with an infrared camera and a visible light camera, performs target detection and instance segmentation on the image data using the YOLOv8 algorithm to construct a mask image set; pairs and processes the mask images of each target of the left and right cameras by constructing an energy loss function to obtain a target pairing and correction image of the left camera target mask image and the right camera target mask image; extracts image feature points of the target pairing and introduces an angle fusion score, screens the extracted feature points to obtain the best feature points, performs image registration on the best feature points based on bit difference, and fuses and displays the registered left and right camera images; calculates the optimal binocular disparity using a collaborative registration dual energy function, calculates the three-dimensional coordinates of the target image in actual space by combining the optimal binocular disparity with the triangulation principle, and constructs a three-dimensional point cloud model of the lifting operation site of the lifting machinery and a target point cloud of personnel; obtains the spatial position of the crane by image matching, and moves the target point cloud image of the personnel to the background point cloud according to the spatial position coordinates of the crane to form a complete real-time personnel monitoring point cloud image. The beneficial effects produced include:

[0028] Improved hoisting safety: By combining heterogeneous images from infrared and visible light cameras, binocular vision technology enables real-time monitoring of hazardous areas during hoisting operations. This method maintains efficient target detection and monitoring capabilities under varying lighting conditions, ensuring real-time safety monitoring of personnel and equipment within the operation area. Through precise target matching and calibration, combined with 3D point cloud modeling technology, it comprehensively monitors hazardous factors within the hoisting operation area, promptly identifies potential risks, and effectively improves operational safety.

[0029] High-precision 3D target positioning: Utilizing binocular parallax and triangulation principles, the system calculates the optimal binocular parallax through collaborative registration, accurately calculating the target's 3D coordinates. This method avoids the spatial positioning accuracy issues associated with traditional monocular vision or single-image registration methods, achieving high-precision 3D positioning of targets at the lifting site. The 3D point cloud model accurately reflects the spatial layout and personnel dynamics of the worksite, further enhancing the safety of personnel and equipment during lifting operations.

[0030] Intelligent and real-time monitoring: Utilizing the YOLOv8 algorithm for efficient target detection and instance segmentation, it can quickly identify and segment targets in complex scenarios, reducing manual intervention and improving monitoring automation. Through image registration, target matching, and image fusion technologies, the precise location of targets within the lifting operation area can be acquired in real time, ensuring real-time monitoring and tracking of operators and equipment. In particular, by combining real-time acquisition of the crane's spatial position, the target point cloud image of the operator can be fused with the crane's background point cloud to form a complete real-time monitoring image, providing data support for subsequent decision-making.

[0031] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. In the drawings:

[0033] Figure 1 This is a flow chart showing a position monitoring method based on heterogeneous image registration and binocular positioning according to an exemplary embodiment of the present invention;

[0034] Figure 2 The figure is a structural diagram of a position monitoring system based on heterogeneous image registration and binocular positioning, showing an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0036] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0037] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0038] Example 1

[0039] A position monitoring method based on heterogeneous image registration and binocular positioning, such as Figure 1 Shown, including:

[0040] By setting up an infrared camera and a visible light camera to form a left and right binocular camera, image data of the dangerous area of ​​the lifting operation is obtained. The YOLOv8 algorithm is used to perform target detection and instance segmentation on the image data to construct a mask image set.

[0041] By constructing an energy loss function, the mask images of each target of the left and right cameras are paired and processed to obtain the target pairing and correction image of the left camera target mask image and the right camera target mask image;

[0042] Extract the target paired image feature points and introduce angle fusion scoring. Filter the extracted feature points to obtain the best feature points. Perform image registration on the best feature points based on bit difference. Fusion display the registered left and right camera images.

[0043] The optimal binocular disparity is calculated using the collaborative registration dual energy function. The three-dimensional coordinates of the target image in real space are calculated by combining the optimal binocular disparity with the triangulation principle to construct a three-dimensional point cloud model of the lifting operation site and the target point cloud of the personnel.

[0044] The spatial position of the crane is obtained through image matching, and the target point cloud image of the personnel is moved to the background point cloud according to the spatial position coordinates of the crane to form a complete real-time personnel monitoring point cloud image.

[0045] The present invention is further configured to use the YOLOv8 algorithm to perform target detection and instance segmentation on the acquired left camera infrared image Image_left and right camera visible light image Image_right to create a target mask contour image, obtain a mask binary image of the target after training mapping, and construct a mask image set mask_left and mask_right, wherein mask_left is the left camera mask image set, and mask_right is the right camera mask image set. Specifically, the visible light images and infrared images taken at the crane site are used as data sets. CVAT is used to annotate the images and output mask data sets and mask images. The constructed training data set is randomly divided into training set, validation set and test set. The YOLOv8 network model is built using the PyTorch framework, and the corresponding network training parameters are set. The network training parameters include model setting, training setting, validation setting and test setting. After the setting is completed, the open source pre-trained model is loaded and training is started using the GPU. After training is completed, the YOLOv8 instance segmentation network model with the best performance in the validation set is obtained. The visible light image and infrared image are respectively input into the trained target detection network, and object recognition information is output, including the object category and detection box location information and target mask information. The contour point information of the target to be detected is obtained through YOLOv8 instance segmentation and mapped to a pre-made pure black image with the same resolution and size as the original image. The part between the contour points is filled with a white image as a polygon, and finally a mask binary image of each target of interest in the image is formed and constructed into a mask image set.

[0046] The present invention is further configured to calculate the energy loss function logic: Loss match =∑(mask_left[a] i,j -mask_right[b] i,j ), where Loss matchis an energy loss function. By calculating the energy loss of each mask image of the left and right cameras, the target mask image mask_right[b] of the right camera that is paired with the target mask image mask_left[a] of the left camera corresponding to the minimum energy loss value is taken to form a target pairing; the paired left and right mask images are corrected using binocular image stereo correction technology to obtain the corrected left mask image rectified_mask_left and right mask image rectified_mask_right. Specifically, if there are multiple target information in the same image, the mask images of each target of the left and right cameras are paired by constructing an energy loss function, and the target mask image set of each category of the left camera is traversed. For each target mask image, the energy loss function Loss is performed with each target mask image in the target mask image set of the same category of the right camera. match Calculate and take the energy loss function Loss match The right camera target mask image corresponding to the minimum value is used as the left camera target mask image. If there is a situation where the same right camera target mask image corresponds to multiple left camera target mask images, the energy loss values ​​of all pairing methods containing the right camera target mask image are compared and the energy loss Loss is taken. match The left camera target mask image with the smallest value is its corresponding paired image, and the other left camera target mask images select the energy loss Loss in the corresponding group match The second smallest right camera target mask image is paired.

[0047] The present invention is further configured to use the ORB operator to extract feature points from the target pairing to obtain candidate feature points f, and the optimal feature point screening logic is: f best =max(S match (f)), where f best is the best feature point; S match (f) is the angle fusion score of the candidate feature points, and the calculation logic is: Where N is the number of feature points; d i is the distance of the i-th matching point; d max is the maximum matching distance allowed; w i is the weight of the i-th matching point; e angle,i is the angle error, calculation logic: in, and is the direction vector of the left and right camera target mask images under the i-th candidate feature point; S localgeo,i Score local geometric consistency, calculation logic: Among them, K is the number of neighborhood points of the i-th candidate feature point; d ijis the distance between the i-th matching point and the j-th matching point; w j is the weight of the jth neighborhood point. Specifically, the best feature point is selected by fusion scoring of the feature points, e angle,i is the angle error, which measures the direction difference between the feature points under two perspectives. It is a distance score used to evaluate the distance accuracy between matching points. The smaller the distance, the better the match and the higher the score. Score for the angle error, used to evaluate the directional consistency between matching points. The smaller the angle error, the stronger the directional consistency and the higher the score. localgeo,i It is a local geometric consistency score used to measure the geometric consistency between feature points and neighborhood points. The higher the local consistency, the higher the score.

[0048] The present invention is further configured to use the Brief operator to calculate the descriptor of the optimal feature point, use the Brute-Force matcher on the obtained feature point and the descriptor, set the matching condition to match based on the bit difference, and obtain the matching result, wherein the bit difference calculation logic is: Among them, D(A,B) is the bit difference, N is the number of bits of the descriptor, A i and B i are the values ​​of the two descriptors at the i-th position, respectively, and ⊕ is the exclusive OR operator. Specifically, for the same paired left and right target mask images, the set of best feature point descriptors detected in the left target mask image is traversed. The descriptor with the smallest distance to each feature point descriptor in the right target mask image is calculated and the best matching point is obtained, forming the matching result.

[0049] The present invention is further configured to cut out the target image from the left camera image according to the result of instance segmentation based on the matching result, and project all target points according to the perspective transformation matrix to obtain the registered left camera target image Image2; the image fusion principle is: Image3(x,y)=αImage2(x,y)+(1-α)Image_right, where Image3 is the fused image, x and y are the horizontal coordinates and vertical coordinates of the image respectively, α is the fusion weight, and the calculation logic is: Among them, G red is the gradient amplitude of the infrared image in the target area, G rgb is the gradient amplitude of the visible light image in the target area, and ∈ is the minimum value to prevent the denominator from being zero. Specifically, the perspective transformation matrix of the left image to the right image registration transformation is calculated based on the matching results, according to the following formula: Among them, x', y' are the coordinates of the feature points of the right camera image, and x, y are the coordinates of the feature points of the left camera image. The specific value of the M matrix is ​​calculated through four sets of matching feature point coordinates. The left camera image is cut out of the target image according to the result of instance segmentation. All target points of the target image are projected according to the perspective transformation matrix to obtain the registered left camera target image Image2. The gradient calculation of the left camera image and the right camera image is performed using the Sobel operator to obtain the fusion weight between the two. Then, the target position of the right camera image is weightedly fused with the registered left camera target image to obtain the visible light and infrared target fusion image Image3.

[0050] The present invention is further configured to calculate the collaborative registration dual energy function logic S(d) = Σ x,y |rectified_mask_left(xd,y)-rectified_mask_right(x,y)|, where d is the pixel distance of translation and S(d) is the co-registration dual energy value; the optimal binocular disparity calculation logic is: Where disp is the optimal binocular disparity. Specifically, for each translation distance d, we calculate the pixel difference between the left image and the right image at the same position after translation. The smaller the difference, the better the match between the two images and the smaller the co-registration dual energy value d. By calculating the energy for different translation distances, we finally select the d that minimizes the co-registration dual energy value as the optimal disparity; S(d) is the co-registration dual energy value, which reflects the total loss when the left camera mask image is translated to the left by a distance d. rectified_mask_left(xd,y) is the pixel value in the rectified left image mask image after translation by d pixels, and rectified_mask_right(x,y) is the pixel value in the rectified right image mask image.

[0051] The present invention is further configured to complete target positioning based on the triangulation principle, and the target positioning calculation logic is: Among them, Location (X, Y, Z) is the target location, cx and cy are the coordinates of the center point in the pixel coordinate system, B is the actual physical distance between the optical centers of the binocular cameras, and f is the focal length of the camera; the crane carries the camera to collect on-site images to form the binocular camera fusion image Image3, and uses Colmap for multi-view stereo modeling to generate a complete 3D point cloud model of the crane lifting operation site; the non-zero pixel values ​​in the rectified left camera mask image rectified_mask_left are filled with the optimal binocular disparity to obtain a disparity image, and the binocular stereo vision principle is used to restore the disparity image to a 3D point cloud image in space. The thickness δ is given according to the empirical value, and the point cloud in the generated plane point cloud image is randomly scattered. The scattered space is within the range of δ before and after the original plane to simulate the thickness of the personnel to obtain the personnel target point cloud. Specifically, the target positioning method is based on the principle of triangulation. Using binocular vision technology, the parallax information between the left and right images is obtained, and the coordinates of the target object in three-dimensional space are then inferred. The angle measurement principle is to calculate the position of an object in space by using the known distance between two observation points and the angle or parallax information obtained from these two observation points. In binocular stereo vision, two cameras capture images of the same scene. The different parallaxes of objects can be used to infer their three-dimensional coordinates based on the geometric relationship of triangles.

[0052] The present invention further utilizes Colmap's image registration function to match the currently captured image with a background image in a database, sets the crane as the target, and obtains the crane's real-time spatial location (Location (X, Y, Z)) using a target positioning method. Based on the crane's spatial position, the target point cloud image of the personnel is mapped onto the background point cloud, thereby enabling real-time monitoring and tracking of personnel. Specifically, Colmap's image registration function is further utilized to compare the live image captured by the current camera with the background image in the database, thereby enabling real-time positioning of the crane at the lifting operation site. The target point cloud of the personnel is then mapped onto the background point cloud, enabling real-time monitoring and tracking of personnel.

[0053] Example 2

[0054] See also Figure 2 , the exemplary position monitoring system based on heterogeneous image registration and binocular positioning includes:

[0055] Image acquisition and target detection module: This module uses an infrared camera and a visible light camera to form a left and right binocular camera to acquire image data of the hazardous area during lifting operations. The YOLOv8 algorithm is used to perform target detection and instance segmentation on the image data to construct a mask image set.

[0056] Target pairing and correction module: This module pairs and processes the mask images of each target in the left and right cameras by constructing an energy loss function to obtain the target pairing and correction images of the left camera target mask image and the right camera target mask image;

[0057] Image registration and fusion module: This module extracts the image feature points of the target pair and introduces angle fusion scoring. The extracted feature points are screened to obtain the best feature points. The best feature points are then registered based on the bit difference, and the registered left and right camera images are fused and displayed.

[0058] 3D space reconstruction and point cloud generation module: This module extracts target paired image feature points and introduces angle fusion scoring. The extracted feature points are screened to obtain the best feature points. The best feature points are then registered based on bit difference, and the registered left and right camera images are fused and displayed.

[0059] Background integration and real-time monitoring module: The spatial position of the crane is obtained through image matching, and the target point cloud image of the personnel is moved to the background point cloud according to the spatial position coordinates of the crane to form a complete real-time personnel monitoring point cloud image.

[0060] It should be noted that the position monitoring system based on heterogeneous image registration and binocular positioning provided in the above embodiment and the position monitoring method based on heterogeneous image registration and binocular positioning provided in the above embodiment are of the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here. In actual applications, the position monitoring system based on heterogeneous image registration and binocular positioning provided in the above embodiment can, as needed, allocate the above functions to different functional modules, that is, divide the internal structure of the system into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0061] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0062] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0063] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0064] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0065] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0066] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0067] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0068] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0069] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0070] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0071] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A position monitoring method based on heterogeneous image registration and binocular positioning, characterized in that: include: By setting up an infrared camera and a visible light camera to form a left and right binocular camera, image data of the dangerous area of ​​the lifting operation is obtained. The YOLOv8 algorithm is used to perform target detection and instance segmentation on the image data to construct a mask image set. By constructing an energy loss function, the mask images of each target of the left and right cameras are paired and processed to obtain the target pairing and correction image of the left camera target mask image and the right camera target mask image; Extract the target paired image feature points and introduce angle fusion scoring. Filter the extracted feature points to obtain the best feature points. Perform image registration on the best feature points based on bit difference. Fusion display the registered left and right camera images. The optimal binocular disparity is calculated using the collaborative registration dual energy function. The three-dimensional coordinates of the target image in real space are calculated by combining the optimal binocular disparity with the triangulation principle to construct a three-dimensional point cloud model of the lifting operation site and the target point cloud of the personnel. The spatial position of the crane is obtained through image matching, and the target point cloud image of the personnel is moved to the background point cloud according to the spatial position coordinates of the crane to form a complete real-time personnel monitoring point cloud image.

2. The position monitoring method based on heterogeneous image registration and binocular positioning according to claim 1, characterized in that: The YOLOv8 algorithm is used to perform target detection and instance segmentation on the acquired left camera infrared image Image_left and right camera visible light image Image_right to create a target mask contour image. After training mapping, the target mask binary image is obtained, and the mask image sets mask_left and mask_right are constructed, where mask_left is the left camera mask image set and mask_right is the right camera mask image set.

3. The position monitoring method based on heterogeneous image registration and binocular positioning according to claim 2 is characterized in that: Energy loss function calculation logic: Loss match =∑(mask_left[a] i,j -mask_right[b] i,j ), Among them, Loss match is an energy loss function. The energy loss is calculated for each mask image of the left and right cameras. The right camera target mask image mask_right[b] paired with the left camera target mask image mask_left[a] when the energy loss value is minimized is taken to form a target pair. The paired left and right mask images are corrected using binocular image stereo rectification technology to obtain the corrected left mask image rectified_mask_left and right mask image rectified_mask_right.

4. The position monitoring method based on heterogeneous image registration and binocular positioning according to claim 1, characterized in that: Use the ORB operator to extract feature points from the target pairing and obtain candidate feature points f. The best feature point screening logic is: f best =max(S match (f)), where f best is the best feature point; S match (f) is the angle fusion score of the candidate feature points, and the calculation logic is: Where N is the number of feature points; d i is the distance of the i-th matching point; d max is the maximum matching distance allowed; w i is the weight of the i-th matching point; e angle,i is the angle error, calculation logic: in, and is the direction vector of the left and right camera target mask images under the i-th candidate feature point; S local geo,i Score local geometric consistency, calculation logic: Among them, K is the number of neighborhood points of the i-th candidate feature point; d ij is the distance between the i-th matching point and the j-th matching point; w j is the weight of the j-th neighborhood point.

5. The position monitoring method based on heterogeneous image registration and binocular positioning according to claim 1, characterized in that: The Brief operator is used to calculate the descriptor of the optimal feature point. The Brute-Force matcher is used for the obtained feature points and descriptors. The matching condition is set to match based on the bit difference to obtain the matching result. The bit difference calculation logic is as follows: Among them, D(A,B) is the bit difference, N is the number of bits of the descriptor, A i and B i are the values ​​of the two descriptors at the i-th position, is the exclusive OR operator.

6. The position monitoring method based on heterogeneous image registration and binocular positioning according to claim 1, characterized in that: According to the matching results, the left camera image is cut out into the target image according to the instance segmentation results, and all target points are projected according to the perspective transformation matrix to obtain the registered left camera target image Image2; Image fusion principle: Image3(x,y)=αImage2(x,y)+(1-α)Image_right, where Image3 is the fused image, x and y are the horizontal and vertical coordinates of the image respectively, and α is the fusion weight. The calculation logic is: Among them, G red is the gradient amplitude of the infrared image in the target area, G rgb is the gradient amplitude of the visible light image in the target area, and ∈ is the minimum value to prevent the denominator from being zero.

7. The position monitoring method based on heterogeneous image registration and binocular positioning according to claim 6, characterized in that: Co-registration dual energy function calculation logic S(d) = ∑ x,y |rectified_mask_left(xd,y)-rectified_mask_right(x,y)|, where d is the pixel distance of translation and S(d) is the co-registration dual energy value; the optimal binocular disparity calculation logic is: Among them, disp is the optimal binocular disparity.

8. The position monitoring method based on heterogeneous image registration and binocular positioning according to claim 7, characterized in that: Target positioning is completed based on the principle of triangulation. The target positioning calculation logic is: Among them, Location (X, Y, Z) is the target location, cx and cy are the coordinates of the center point in the pixel coordinate system, B is the actual physical distance between the optical centers of the binocular cameras, and f is the focal length of the camera; the crane carries the camera to collect on-site images to form the binocular camera fusion image Image3, and uses Colmap for multi-view stereo modeling to generate a complete 3D point cloud model of the crane lifting operation site; the non-zero pixel values ​​in the rectified left camera mask image rectified_mask_left are filled with the optimal binocular disparity to obtain a disparity image, and the binocular stereo vision principle is used to restore the disparity image to a 3D point cloud image in space. The thickness δ is given according to the empirical value, and the point cloud in the generated plane point cloud image is randomly scattered. The scattered space is within the range of δ before and after the original plane to simulate the thickness of the personnel to obtain the personnel target point cloud.

9. The position monitoring method based on heterogeneous image registration and binocular positioning according to claim 8, characterized in that: The image registration function of Colmap is used to match the currently captured image with the background image in the database. The crane is set as the target, and the real-time spatial position (Location (X, Y, Z)) of the crane is obtained according to the target positioning method. Based on the spatial position of the crane, the target point cloud image of the personnel is mapped to the background point cloud to achieve real-time monitoring and tracking of personnel.

10. A position monitoring system based on heterogeneous image registration and binocular positioning, used to implement the position monitoring method based on heterogeneous image registration and binocular positioning according to any one of claims 1 to 9, characterized in that: include: Image acquisition and target detection module: This module uses an infrared camera and a visible light camera to form a left and right binocular camera to acquire image data of the hazardous area during lifting operations. The YOLOv8 algorithm is used to perform target detection and instance segmentation on the image data to construct a mask image set. Target pairing and correction module: This module pairs and processes the mask images of each target in the left and right cameras by constructing an energy loss function to obtain the target pairing and correction images of the left camera target mask image and the right camera target mask image; Image registration and fusion module: This module extracts the image feature points of the target pair and introduces angle fusion scoring. The extracted feature points are screened to obtain the best feature points. The best feature points are then registered based on the bit difference, and the registered left and right camera images are fused and displayed. 3D space reconstruction and point cloud generation module: This module extracts target paired image feature points and introduces angle fusion scoring. The extracted feature points are screened to obtain the best feature points. The best feature points are then registered based on bit difference, and the registered left and right camera images are fused and displayed. Background integration and real-time monitoring module: The spatial position of the crane is obtained through image matching, and the target point cloud image of the personnel is moved to the background point cloud according to the spatial position coordinates of the crane to form a complete real-time personnel monitoring point cloud image.

Images