Gun and ball linkage three-dimensional engine, server, system and method

Abstract

The invention relates to the technical field of image monitoring, and discloses a three-dimensional engine, a server, a system and a method for gun and ball linkage, and the three-dimensional engine is used for executing the following steps: when a mouse clicks a three-dimensional scene, obtaining a pixel coordinate of a mouse clicking position in a curtain binding image; transmitting the bolt information bound with the curtain and the pixel coordinates of the mouse click position to a server; according to the PT value of the dome camera bound with the gun camera, the dome camera is called to rotate to the direction of shooting the image of the mouse click position; the server is used for executing the following steps: receiving bolt information bound with a curtain and pixel coordinates of a mouse click position sent by a three-dimensional engine; calculating a PT value of the dome camera bound with the gun camera through the homography matrix configuration file; and returning the PT value of the dome camera bound with the gun camera to the three-dimensional engine. According to the invention, the problems of low gun-ball linkage accuracy and the like in an environment which is not easy to calibrate in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of image surveillance technology, specifically a 3D engine, server, system, and method for camera-ball linkage. Background Technology

[0002] Camera-PTZ linkage is an image monitoring technology that achieves seamless integration of panoramic monitoring and detailed tracking through the coordinated work of a bullet camera (fixed wide-angle lens) and a PTZ camera (rotatable zoom lens supporting rapid positioning). Its core principle is: the bullet camera is responsible for detecting targets over a wide area; when a rule is triggered (such as area intrusion), the PTZ camera automatically turns to the target location for close-up tracking; after the target leaves, the PTZ camera returns to its preset position.

[0003] Currently, most bullet-and-PTZ camera linkage solutions involve one bullet camera linked to one PTZ camera, and this is primarily achieved through clicking on a two-dimensional pixel screen. Calibration of both bullet and PTZ cameras requires a checkerboard pattern to determine internal and external parameters, which is unsuitable for outdoor environments or older equipment where calibration is difficult. This is mainly reflected in the following aspects: (1) Technical aspects: Traditional high-precision calibration methods are mainly based on Zhang Zhengyou's calibration method. Its accuracy is extremely high and it is the gold standard for verifying the accuracy of other methods, but it requires the production of high-precision three-dimensional calibration objects, which is costly and inconvenient to use.

[0004] (2) A sufficient number of images (about 15) with rich pose variations (covering the entire field of view and having various rotation angles) must be collected. Therefore, it is not easy to calibrate in poor environments such as outdoors.

[0005] Therefore, under this background, the existing technology has the following problems: calibration is troublesome in environments where it is not easy to calibrate, it is difficult to achieve multi-ball linkage with one gun, and it is difficult to achieve three-dimensional spatial point linkage PTZ camera, resulting in low accuracy of gun-ball linkage. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a 3D engine, server, system, and method for gun-ball linkage, solving problems such as low accuracy of gun-ball linkage in environments that are difficult to calibrate.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: A 3D engine for gun-ball linkage, used to perform the following steps: When the mouse clicks on the 3D scene, the mouse click action is captured, and it is determined whether the mouse click position falls within the screen: if yes, proceed to the next step; otherwise, no action is taken. Get the pixel coordinates of the mouse click position in the image bound to the screen; where the image bound to the screen refers to the image captured by the camera attached to the screen. Send the camera information bound to the screen and the pixel coordinates of the mouse click position to the server; When the server returns the PT value of the PT camera bound to the bullet camera, the PT value of the PT camera bound to the bullet camera is received; where the PT value refers to the horizontal rotation value and the vertical rotation value. The camera's PT value is used to rotate the PT camera to the direction where the image at the mouse click position is captured.

[0008] The beneficial effects of this invention are: This invention's 3D engine acquires the camera information and pixel coordinates of the 3D image at the mouse click location and sends it to the server. When the server returns the PT value of the PT camera bound to the camera, it receives the PT value and then uses the PT camera to capture the mouse click location. This eliminates the need for costly high-precision 3D calibration objects, is convenient to use, and avoids the need to acquire a large number of images with diverse poses, or images covering the entire field of view with various rotation angles. Calibration is convenient in outdoor environments or with older equipment where calibration is difficult. It facilitates multi-PT camera linkage and 3D spatial point linkage, resulting in high accuracy for camera-PT camera linkage.

[0009] Based on the above technical solution, the present invention can be further improved as follows.

[0010] As a preferred technical solution, obtaining the pixel coordinates of the mouse click position in the image bound to the backdrop includes the following steps: Project the 3D image at the mouse click location onto a screen to generate a 2D image; Based on the spatial coordinates of the gun, the direction of illumination, and the projection distance, the texture coordinates of the two-dimensional image are calculated through coordinate transformation. Based on the texture coordinates of the 2D image, the pixel coordinates of the mouse click position in the image bound to the screen are obtained by mapping the resolution of the 3D image.

[0011] The beneficial effects of adopting the above-mentioned preferred technical solution are: Obtaining the pixel coordinates of the mouse click position in the image bound to the screen through 3D image projection and its geometric and quantitative relationships yields high accuracy.

[0012] As a preferred technical solution, the step of calling the PT value of the PT camera bound to the bullet camera to rotate the PT camera to the direction of capturing the image at the mouse click position includes the following steps: the 3D engine uses the setptz interface to call the PT camera to rotate the PT camera to the direction of capturing the image at the mouse click position based on the PT value of the PT camera bound to the bullet camera.

[0013] The beneficial effects of adopting the above-mentioned preferred technical solution are: Using the setptz interface, the camera can be rotated to the direction of the PT value of the PT camera bound to the bullet camera to capture the image at the mouse click position. This operation is convenient and efficient.

[0014] The present invention also provides a server for gun-ball linkage.

[0015] A server for gun-ball linkage, used to perform the following steps: Receive the camera information bound to the screen and the pixel coordinates of the mouse click position sent by the 3D engine; Based on the camera information bound to the screen and the pixel coordinates of the mouse click position, the PT value of the PT camera bound to the screen is calculated through the homography matrix configuration file. The PT value of the PT camera bound to the bullet camera is returned to the 3D engine.

[0016] The beneficial effects of this invention are: This invention's server receives the camera information bound to the screen and the pixel coordinates of the mouse click position from the 3D engine. It calculates the PT value of the PT camera bound to the screen using a homography matrix configuration file, and then returns the PT value of the PT camera to the 3D engine. This eliminates the need for costly high-precision 3D calibration objects, making it convenient to use. It avoids the need to acquire a large number of images with diverse poses, or images covering the entire field of view with various rotation angles. Calibration is convenient in outdoor environments or with older equipment where calibration is difficult. It facilitates one-camera-multiple-PTZ linkage and 3D spatial point-to-PTZ linkage, resulting in high accuracy in camera-PTZ linkage.

[0017] Based on the above technical solution, the present invention can be further improved as follows.

[0018] As a preferred technical solution, the step-to-body (PT) value of the PT camera bound to the projector is calculated using a homography matrix configuration file based on the projector information bound to the screen and the pixel coordinates of the mouse click position. This includes the following steps: Load the homography matrix configuration file; the homography matrix configuration file includes the homography matrix of the bullet camera and the PTZ camera, and the configuration information of the PTZ camera bound to the bullet camera; Based on the camera information bound to the screen and the pixel coordinates of the mouse click position, obtain the homography matrix of the camera and the PT camera in the homography matrix configuration file, as well as the configuration information of the PT camera bound to the camera, and calculate the PT value of the PT camera bound to the camera.

[0019] The beneficial effects of adopting the above-mentioned preferred technical solution are: The information in the homography matrix configuration file makes it easy to calculate the PT value of the PT camera bound to the bullet camera in a timely manner, resulting in a fast response speed.

[0020] As a preferred technical solution, the homography matrix configuration file includes: a homography matrix and configuration information for a case where one camera is bound to one PTZ camera, and a homography matrix and configuration information for a case where one camera is bound to two or more PTZ cameras.

[0021] The beneficial effects of adopting the above-mentioned preferred technical solution are: This invention is applicable to situations involving multiple balls with one gun or one ball with one gun, thus having a wide range of applications.

[0022] As a preferred technical solution, obtaining the homography matrix of the bullet camera and the PTZ camera in the homography matrix configuration file includes the following steps: The homography matrices of the bullet and PTZ cameras in the homography matrix configuration file are obtained by calling the findHomography function.

[0023] The beneficial effects of adopting the above-mentioned preferred technical solution are: It facilitates quick and easy acquisition of homography matrices and consumes minimal computational resources.

[0024] Based on the above technical solutions, the present invention also provides a gun-ball linkage system.

[0025] A ball-and-gun linkage system includes a ball-and-gun linkage 3D engine and a ball-and-gun linkage server connected to the ball-and-gun linkage 3D engine.

[0026] The beneficial effects of this invention are: This invention's 3D engine acquires the camera information and pixel coordinates of the 3D image at the mouse click location and sends it to the server. When the server returns the PT value of the PT camera bound to the camera, the engine receives the PT value and then uses the PT camera to capture the mouse click location. The server receives the camera information bound to the screen and the pixel coordinates of the mouse click location from the 3D engine, calculates the PT value of the PT camera bound to the camera using a homography matrix configuration file, and then returns the PT value of the PT camera bound to the camera to the 3D engine. This invention eliminates the need for costly high-precision 3D calibration objects, is convenient to use, and avoids acquiring a large number of images with diverse poses, or images covering the entire field of view with various rotation angles. Calibration is convenient in outdoor environments or with older equipment where calibration is difficult. It facilitates one-camera-multiple-PT camera linkage and 3D spatial point-to-PT camera linkage, resulting in high accuracy in camera-PT camera linkage.

[0027] Based on the above technical solution, the present invention can be further improved as follows.

[0028] As a preferred technical solution, it also includes a calibration tool connected to the aforementioned gun-ball linkage server, used to perform the following steps: Calculate the homography matrix of the camera and the PTZ camera based on the pixel coordinates of the camera and the matching pair of the camera and the PTZ camera; The homography matrix of the bullet camera and the PTZ camera, and the configuration information of the PTZ camera bound to the bullet camera are saved as a homography matrix configuration file; Send the homography matrix configuration file to the server; The methods for generating the matching pairs between the camera pixel coordinates and the PTZ camera include one or two of the following: Manual calibration: Manual calibration generates matching pairs between bullet camera pixel coordinates and PT values ​​of the PT camera; Automatic calibration includes the following steps: The PT camera automatically scans images and saves them; each image corresponds to a PT value. The image saved by the PTZ camera is matched with the image saved by the bullet camera, and the matching is determined according to a set threshold. If the matching is successful, the pixel coordinates of the bullet camera that matched the image saved by the PTZ camera are calculated. The successfully matched PT camera information and bullet camera pixel coordinates are used as matching pairs between bullet camera pixel coordinates and PT camera values.

[0029] The beneficial effects of adopting the above-mentioned preferred technical solution are: The calibration tool of this invention can generate matching pairs between the camera pixel coordinates and the PTZ camera through manual or automatic calibration. Then, it calculates the homography matrix of the camera and the PTZ camera based on the matching pairs. Finally, it saves the homography matrix of the camera and the PTZ camera, as well as the configuration information of the PTZ camera bound to the camera, as a homography matrix configuration file and sends it to the server. The operation is convenient and also makes it easy for the server to quickly and accurately obtain the PT value of the PTZ camera bound to the camera, thereby further improving the efficiency and accuracy of the camera-PTZ linkage.

[0030] Based on the above technical solutions, the present invention also provides a gun-ball linkage method.

[0031] A method for linking gun and ball, using the aforementioned gun-ball linkage system, includes the following steps: When the mouse clicks on the 3D scene, the 3D engine receives the mouse click action and determines whether the mouse click position falls within the screen: if yes, it proceeds to the next step; otherwise, it does nothing. The 3D engine obtains the pixel coordinates of the mouse click position in the image bound to the screen; where the image bound to the screen refers to the image captured by the camera attached to the screen. The 3D engine sends the camera information bound to the screen and the pixel coordinates of the mouse click position to the server; The server receives the camera information bound to the screen and the pixel coordinates of the mouse click position from the 3D engine. Based on the camera information bound to the screen and the pixel coordinates of the mouse click position, the server calculates the PT value of the PTZ camera bound to the screen using the homography matrix configuration file. The server returns the PT value of the PT camera bound to the bullet camera to the 3D engine; The 3D engine receives the PT value of the PT camera attached to the bullet camera; where PT value refers to the horizontal rotation value and the vertical rotation value. The 3D engine uses the PT value of the PT camera attached to the bullet camera to rotate the PT camera to capture the image at the mouse click position.

[0032] Compared with the prior art, the present invention has the following advantages: (1) The 3D engine of this invention obtains the camera information and pixel coordinates of the 3D image of the mouse click position and sends it to the server. When the server returns the PT value of the PT camera bound to the camera, it receives the PT value of the PT camera bound to the camera and then calls the PT camera to capture the mouse click position. The server of this invention receives the camera information bound to the screen and the pixel coordinates of the mouse click position sent by the 3D engine, calculates the PT value of the PT camera bound to the camera through the homography matrix configuration file, and then returns the PT value of the PT camera bound to the camera to the 3D engine. It does not require the high cost of making high-precision 3D calibration objects, is convenient to use, does not need to collect a large number of images with rich posture changes, and does not need to collect images covering the entire field of view with various rotation angles. It is convenient to calibrate in some outdoor, old equipment and other environments where calibration is not easy, and facilitates the realization of one camera and multiple PT cameras, and 3D spatial point linkage PT cameras, thereby making the accuracy of camera-PT camera linkage high.

[0033] (2) The calibration tool of this invention can generate the matching pairs of the bullet camera pixel coordinates and the PT camera through manual calibration and automatic calibration. Then, it calculates the homography matrix of the bullet camera and the PT camera based on the matching pairs. The homography matrix of the bullet camera and the PT camera and the configuration information of the PT camera bound to the bullet camera are saved as a homography matrix configuration file and sent to the server. The operation is convenient and the server can quickly and accurately obtain the PT value of the PT camera bound to the bullet camera, thereby further improving the efficiency of the bullet camera linkage and the accuracy of the bullet camera linkage. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the working process of a gun-ball linkage system according to the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0036] The principles and features of the present invention are described below. The embodiments given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0037] Example 1 like Figure 1 As shown, a 3D engine for gun-ball linkage is used to perform the following steps: When the mouse clicks on the 3D scene, the mouse click action is captured, and it is determined whether the mouse click position falls within the screen: if yes, proceed to the next step; otherwise, no action is taken. Get the pixel coordinates of the mouse click position in the image bound to the screen; where the image bound to the screen refers to the image captured by the camera attached to the screen. Send the camera information bound to the screen and the pixel coordinates of the mouse click position to the server; When the server returns the PT value of the PT camera bound to the bullet camera, the PT value of the PT camera bound to the bullet camera is received; where the PT value refers to the horizontal rotation value and the vertical rotation value. The camera's PT value is used to rotate the PT camera to the direction where the image at the mouse click position is captured.

[0038] The beneficial effects of this invention are: This invention's 3D engine acquires the camera information and pixel coordinates of the 3D image at the mouse click location and sends it to the server. When the server returns the PT value of the PT camera bound to the camera, it receives the PT value and then uses the PT camera to capture the mouse click location. This eliminates the need for costly high-precision 3D calibration objects, is convenient to use, and avoids the need to acquire a large number of images with diverse poses, or images covering the entire field of view with various rotation angles. Calibration is convenient in outdoor environments or with older equipment where calibration is difficult. It facilitates multi-PT camera linkage and 3D spatial point linkage, resulting in high accuracy for camera-PT camera linkage.

[0039] Based on the above technical solution, the present invention can be further improved as follows.

[0040] As a preferred technical solution, obtaining the pixel coordinates of the mouse click position in the image bound to the backdrop includes the following steps: Project the 3D image at the mouse click location onto a screen to generate a 2D image; Based on the spatial coordinates of the gun, the direction of illumination, and the projection distance, the texture coordinates of the two-dimensional image are calculated through coordinate transformation. Based on the texture coordinates of the 2D image, the pixel coordinates of the mouse click position in the image bound to the screen are obtained by mapping the resolution of the 3D image.

[0041] The beneficial effects of adopting the above-mentioned preferred technical solution are: Obtaining the pixel coordinates of the mouse click position in the image bound to the screen through 3D image projection and its geometric and quantitative relationships yields high accuracy.

[0042] As a preferred technical solution, the step of calling the PT value of the PT camera bound to the bullet camera to rotate the PT camera to the direction of capturing the image at the mouse click position includes the following steps: the 3D engine uses the setptz interface to call the PT camera to rotate the PT camera to the direction of capturing the image at the mouse click position based on the PT value of the PT camera bound to the bullet camera.

[0043] The beneficial effects of adopting the above-mentioned preferred technical solution are: Using the setptz interface, the camera can be rotated to the direction of the PT value of the PT camera bound to the bullet camera to capture the image at the mouse click position. This operation is convenient and efficient.

[0044] The present invention also provides a server for gun-ball linkage.

[0045] A server for gun-ball linkage, used to perform the following steps: Receive the camera information bound to the screen and the pixel coordinates of the mouse click position sent by the 3D engine; Based on the camera information bound to the screen and the pixel coordinates of the mouse click position, the PT value of the PT camera bound to the screen is calculated through the homography matrix configuration file. The PT value of the PT camera bound to the bullet camera is returned to the 3D engine.

[0046] The beneficial effects of this invention are: This invention's server receives the camera information bound to the screen and the pixel coordinates of the mouse click position from the 3D engine. It calculates the PT value of the PT camera bound to the screen using a homography matrix configuration file, and then returns the PT value of the PT camera to the 3D engine. This eliminates the need for costly high-precision 3D calibration objects, making it convenient to use. It avoids the need to acquire a large number of images with diverse poses, or images covering the entire field of view with various rotation angles. Calibration is convenient in outdoor environments or with older equipment where calibration is difficult. It facilitates one-camera-multiple-PTZ linkage and 3D spatial point-to-PTZ linkage, resulting in high accuracy in camera-PTZ linkage.

[0047] Based on the above technical solution, the present invention can be further improved as follows.

[0048] As a preferred technical solution, the step-to-body (PT) value of the PT camera bound to the projector is calculated using a homography matrix configuration file based on the projector information bound to the screen and the pixel coordinates of the mouse click position. This includes the following steps: Load the homography matrix configuration file; the homography matrix configuration file includes the homography matrix of the bullet camera and the PTZ camera, and the configuration information of the PTZ camera bound to the bullet camera; Based on the camera information bound to the screen and the pixel coordinates of the mouse click position, obtain the homography matrix of the camera and the PT camera in the homography matrix configuration file, as well as the configuration information of the PT camera bound to the camera, and calculate the PT value of the PT camera bound to the camera.

[0049] The beneficial effects of adopting the above-mentioned preferred technical solution are: The information in the homography matrix configuration file makes it easy to calculate the PT value of the PT camera bound to the bullet camera in a timely manner, resulting in a fast response speed.

[0050] As a preferred technical solution, the homography matrix configuration file includes: a homography matrix and configuration information for a case where one camera is bound to one PTZ camera, and a homography matrix and configuration information for a case where one camera is bound to two or more PTZ cameras.

[0051] The beneficial effects of adopting the above-mentioned preferred technical solution are: This invention is applicable to situations involving multiple balls with one gun or one ball with one gun, thus having a wide range of applications.

[0052] As a preferred technical solution, obtaining the homography matrix of the bullet camera and the PTZ camera in the homography matrix configuration file includes the following steps: The homography matrices of the bullet and PTZ cameras in the homography matrix configuration file are obtained by calling the findHomography function.

[0053] The beneficial effects of adopting the above-mentioned preferred technical solution are: It facilitates quick and easy acquisition of homography matrices and consumes minimal computational resources.

[0054] Based on the above technical solutions, the present invention also provides a gun-ball linkage system.

[0055] A ball-and-gun linkage system includes a ball-and-gun linkage 3D engine and a ball-and-gun linkage server connected to the ball-and-gun linkage 3D engine.

[0056] The beneficial effects of this invention are: This invention's 3D engine acquires the camera information and pixel coordinates of the 3D image at the mouse click location and sends it to the server. When the server returns the PT value of the PT camera bound to the camera, the engine receives the PT value and then uses the PT camera to capture the mouse click location. The server receives the camera information bound to the screen and the pixel coordinates of the mouse click location from the 3D engine, calculates the PT value of the PT camera bound to the camera using a homography matrix configuration file, and then returns the PT value of the PT camera bound to the camera to the 3D engine. This invention eliminates the need for costly high-precision 3D calibration objects, is convenient to use, and avoids acquiring a large number of images with diverse poses, or images covering the entire field of view with various rotation angles. Calibration is convenient in outdoor environments or with older equipment where calibration is difficult. It facilitates one-camera-multiple-PT camera linkage and 3D spatial point-to-PT camera linkage, resulting in high accuracy in camera-PT camera linkage.

[0057] Based on the above technical solution, the present invention can be further improved as follows.

[0058] As a preferred technical solution, it also includes a calibration tool connected to the aforementioned gun-ball linkage server, used to perform the following steps: Calculate the homography matrix of the camera and the PTZ camera based on the pixel coordinates of the camera and the matching pair of the camera and the PTZ camera; The homography matrix of the bullet camera and the PTZ camera, and the configuration information of the PTZ camera bound to the bullet camera are saved as a homography matrix configuration file; Send the homography matrix configuration file to the server; The methods for generating the matching pairs between the camera pixel coordinates and the PTZ camera include one or two of the following: Manual calibration: Manual calibration generates matching pairs between bullet camera pixel coordinates and PT values ​​of the PT camera; Automatic calibration includes the following steps: The PT camera automatically scans images and saves them; each image corresponds to a PT value. The image saved by the PTZ camera is matched with the image saved by the bullet camera, and the matching is determined according to a set threshold. If the matching is successful, the pixel coordinates of the bullet camera that matched the image saved by the PTZ camera are calculated. The successfully matched PT camera information and bullet camera pixel coordinates are used as matching pairs between bullet camera pixel coordinates and PT camera values.

[0059] The beneficial effects of adopting the above-mentioned preferred technical solution are: The calibration tool of this invention can generate matching pairs between the camera pixel coordinates and the PTZ camera through manual or automatic calibration. Then, it calculates the homography matrix of the camera and the PTZ camera based on the matching pairs. Finally, it saves the homography matrix of the camera and the PTZ camera, as well as the configuration information of the PTZ camera bound to the camera, as a homography matrix configuration file and sends it to the server. The operation is convenient and also makes it easy for the server to quickly and accurately obtain the PT value of the PTZ camera bound to the camera, thereby further improving the efficiency and accuracy of the camera-PTZ linkage.

[0060] Based on the above technical solutions, the present invention also provides a gun-ball linkage method.

[0061] A method for linking gun and ball, using the aforementioned gun-ball linkage system, includes the following steps: When the mouse clicks on the 3D scene, the 3D engine receives the mouse click action and determines whether the mouse click position falls within the screen: if yes, it proceeds to the next step; otherwise, it does nothing. The 3D engine obtains the pixel coordinates of the mouse click position in the image bound to the screen; where the image bound to the screen refers to the image captured by the camera attached to the screen. The 3D engine sends the camera information bound to the screen and the pixel coordinates of the mouse click position to the server; The server receives the camera information bound to the screen and the pixel coordinates of the mouse click position from the 3D engine. Based on the camera information bound to the screen and the pixel coordinates of the mouse click position, the server calculates the PT value of the PTZ camera bound to the screen using the homography matrix configuration file. The server returns the PT value of the PT camera bound to the bullet camera to the 3D engine; The 3D engine receives the PT value of the PT camera attached to the bullet camera; where PT value refers to the horizontal rotation value and the vertical rotation value. The 3D engine uses the PT value of the PT camera attached to the bullet camera to rotate the PT camera to capture the image at the mouse click position.

[0062] Example 2 like Figure 1 As shown, based on Example 1, this example provides a more detailed implementation method.

[0063] A gun-ball linkage method, using a gun-ball linkage system (including interconnected 3D engines and servers) to perform gun-ball linkage, includes the following steps: When the mouse clicks on the 3D scene, the 3D engine receives the mouse click action and determines whether the mouse click position falls within the screen: if yes, it proceeds to the next step; otherwise, it does nothing. The 3D engine obtains the pixel coordinates of the mouse click position in the image bound to the screen; where the image bound to the screen refers to the image captured by the camera attached to the screen. The 3D engine sends the camera information bound to the screen and the pixel coordinates of the mouse click position to the server; The server receives the camera information bound to the screen and the pixel coordinates of the mouse click position from the 3D engine. Based on the camera information bound to the screen and the pixel coordinates of the mouse click position, the server calculates the PT value of the PTZ camera bound to the screen using the homography matrix configuration file. The server returns the PT value of the PT camera bound to the bullet camera to the 3D engine; The 3D engine receives the PT value of the PT camera attached to the bullet camera; where PT value refers to the horizontal rotation value and the vertical rotation value. The 3D engine uses the PT value of the PT camera attached to the bullet camera to rotate the PT camera to capture the image at the mouse click position.

[0064] Preferably, in this embodiment, the gun-ball linkage system further includes a calibration tool connected to the server, used to perform the following steps: Calculate the homography matrix of the camera and the PTZ camera based on the pixel coordinates of the camera and the matching pair of the camera and the PTZ camera; The homography matrix of the bullet camera and the PTZ camera, and the configuration information of the PTZ camera bound to the bullet camera are saved as a homography matrix configuration file; Send the homography matrix configuration file to the server; The methods for generating the matching pairs between the camera pixel coordinates and the PTZ camera include one or two of the following: Manual calibration: Manual calibration generates matching pairs between bullet camera pixel coordinates and PT values ​​of the PT camera; Automatic calibration includes the following steps: The PT camera automatically scans images and saves them; each image corresponds to a PT value. The image saved by the PTZ camera is matched with the image saved by the bullet camera, and the matching is determined according to a set threshold. If the matching is successful, the pixel coordinates of the bullet camera that matched the image saved by the PTZ camera are calculated. The successfully matched PT camera information and bullet camera pixel coordinates are used as matching pairs between bullet camera pixel coordinates and PT camera values.

[0065] The more specific technical solutions are as follows: 3D Click Event: When the mouse clicks on a 3D scene, the system receives the click action and internally determines whether the mouse position falls within the screen. If it does, subsequent architectural operations are triggered.

[0066] The method for obtaining image pixel coordinates in a 3D scene is as follows: In a 3D scene, images are usually presented on a screen, which is a geometric shape. After the mouse clicks and projects the spatial image onto the geometric shape of the screen, the texture coordinate position can be calculated through coordinate transformation. After obtaining the texture coordinate position, it can be mapped to the image pixel coordinates.

[0067] Preferably, in this embodiment, the 3D engine obtains the pixel coordinates of the mouse click position in the image bound to the screen through a fusion algorithm.

[0068] The specific details of the fusion algorithm include: 1. By using the spatial position coordinates of the gun and the illumination direction of the camera, a projection matrix of the spatial image can be formed. Based on the focal length and field of view of the camera and the projection distance d from the center projection line of the camera to the 3D scene space, a projection is performed on the scene to produce a projection triangular mesh model (data patch - rectangular patch skinning to the concave and convex scene model surface of the 3D scene). Each spatial 3D point of this triangular mesh model (these points will not exceed the boundary of the image rectangular view cone) corresponds to an image pixel coordinate (U,V), which is also the projection pixel point of the image. The values ​​of U and V (0~1) are the ratio of the pixel coordinate X-axis and Y-axis. 2. Clicking the screen with the mouse causes a ray emanating from the viewpoint to intersect with the image projection triangular mesh model in space. The intersection point is a plane intersection point of a triangular region of the triangular mesh model (bump skin). This intersection point is calculated using the coordinates of the three vertices of the triangle (and the texture UV coordinates of the vertex matching) and bilinear difference. Then, based on the image resolution, the corresponding pixel coordinates are calculated.

[0069] The calculation method for PT value (P: Pan, horizontal rotation or gimbal horizontal rotation; T: Tilt, vertical rotation or gimbal vertical pitch) is as follows: The 3D engine (i.e., the client) sends the camera information bound to the screen (in this embodiment, a blended screen) and the currently clicked pixel coordinates to the PTZ (camera linkage) server. Upon receiving the message, the PTZ server looks up the corresponding homography matrix model based on the camera information (in most cases, it's one camera with multiple PTZs, resulting in multiple homography matrices; in other cases, it's one camera with one PTZ, resulting in only one homography matrix). Then, the homography matrix can be converted into the corresponding PT value of the PTZ camera based on the input pixel coordinates. After the calculation is complete, the PTZ server returns the PT values ​​of each PTZ camera to the 3D engine. The 3D engine calls the SETPTZ interface (fast positioning control interface) provided by the camera manufacturer to rotate the PTZ camera to the direction of capturing the image at the mouse click position based on the PT value of the PTZ camera bound to the camera (more specifically: it calls the PTZ scene device SDK fast positioning function interface, inputs the obtained PT value, and then the PTZ camera rotates to the direction of capturing the image at the mouse click position, so that the center position of the image captured by the PTZ camera is the mouse click position of the image captured by the camera).

[0070] A blending screen is a display system that uses multiple projectors to project images and employs edge blending technology to achieve seamless splicing. Its core principle is to overlap the images projected by multiple projectors at their edges, and through geometric correction and brightness attenuation processing, eliminate splicing gaps to form a complete, high-brightness, ultra-large image. The technical principle is edge blending technology: by adjusting the brightness of the overlapping areas (the brightness on the right side of the left projector decreases linearly, while the brightness on the left side of the right projector increases linearly), the brightness of the entire image is made consistent.

[0071] The homography matrix is ​​calculated as follows: The result is calculated from several PT\pixel pairs (PT and pixel coordinate pairs) input by the calibration tool (at least four matching pairs).

[0072] The homography matrix is ​​calculated based on the matching pairs, and the process is as follows: The homography matrix can be obtained using the findHomography function provided by OpenCV.

[0073] OpenCV (Open Source Computer Vision Library) is an open-source computer vision and machine learning software library initiated by Intel in 1999 and now maintained by the non-profit organization OpenCV.org. It supports multiple programming languages ​​(such as C++, Python, and Java) and provides rich functionality for image processing, image analysis, and object detection. The `findHomography` function is the core function in OpenCV used to calculate the homography transformation between two planes; it can solve for a 3×3 matrix describing the projection relationship between the planes by matching feature point pairs.

[0074] More specifically, a method including the following steps can be adopted: Using a SIFT detector, SIFT features are computed on the image, and then feature points and descriptors are computed to create a feature matcher; Perform feature matching on the descriptors and determine whether a match is successful based on the input threshold; Then, extract the feature point coordinates of the two images from the matched image pairs; Then, the homography matrix is ​​calculated based on the matched key points; Calculate the corresponding position of the smaller image in the larger image using the homography matrix; The perspective transformation function is used to find the position of the large image corresponding to the four vertices. Then, the geometric center point is calculated based on the four points. Finally, the pixel coordinates corresponding to the PT value of the current scanned image are obtained, and the required matching pair is obtained.

[0075] More specifically, after a successful match, high-quality point pairs are obtained based on a threshold (high-quality means removing some point pairs that affect accuracy based on the threshold). Then, the homography matrix is ​​calculated based on the point pairs. The four corner points of the PTZ camera image are then converted to the bullet camera image. Finally, the center point of the quadrilateral in the converted bullet camera image is calculated. This center point is the bullet camera pixel coordinates of the PTZ camera image that have been successfully matched.

[0076] The above steps can be summarized as an image search process, that is, finding the location in the camera's view within the PTZ camera view. Specific features include: Feature matching: Use SIFT (create SIFT features) + FLANN (feature matching) + Lowe's ratio test (remove fuzzy matches based on a pre-set threshold); Geometric transformation: The homography matrix describes the projection relationship from one plane to another (the homography matrix is ​​obtained from several pairs of points obtained by feature matching, then the PT camera image boundary is transformed to the bullet camera image coordinates, then it is determined whether it is a quadrilateral and the center point of the quadrilateral is calculated, thus obtaining the pixel value corresponding to the PT value). Robustness: The RANSAC algorithm is used to handle matching outliers; Visualization: Drawing bounding boxes and center points facilitates verification (center point and the pixel coordinates of the current PT); Data persistence: Save coordinate information for subsequent PTZ control.

[0077] As described above, the present invention can be implemented well.

[0078] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0079] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0080] In the description of this invention, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0081] In the description of this invention, although embodiments of the invention have been shown and described herein, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this invention.

[0082] In the description of this invention, all features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A gunball linked three-dimensional engine characterized by, For executing the following steps: When the mouse clicks the three-dimensional scene, the mouse click action is obtained, and it is judged whether the mouse click position falls in the curtain: if yes, the next step is entered; if not, no action is taken; The pixel coordinates of the mouse click position in the curtain binding image are obtained; wherein the curtain binding image refers to the image shot by the curtain binding gun; The curtain binding gun information and the pixel coordinates of the mouse click position are sent to the server; When the server returns the PT value of the gun binding pan-tilt camera, the PT value of the gun binding pan-tilt camera is received; wherein the PT value refers to the horizontal rotation value and the vertical rotation value; The pan-tilt camera is called to rotate to the direction of shooting the image of the mouse click position according to the PT value of the gun binding pan-tilt camera.

2. A gun-ball linked three-dimensional engine according to claim 1, wherein, The pixel coordinates of the mouse click position in the curtain binding image are obtained, including the following steps: The three-dimensional image of the mouse click position is projected onto the curtain to generate a two-dimensional picture; Based on the spatial position coordinates, the illumination direction, and the projection distance of the gun, the texture coordinate position of the two-dimensional picture is calculated through coordinate conversion; Based on the texture coordinate position of the two-dimensional picture, the pixel coordinates of the mouse click position in the curtain binding image are obtained through three-dimensional image resolution mapping.

3. A gun-ball linked three-dimensional engine according to claim 1 or 2, wherein, The pan-tilt camera is called to rotate to the direction of shooting the image of the mouse click position according to the PT value of the gun binding pan-tilt camera, including the following steps: the three-dimensional engine uses the setptz interface to call the pan-tilt camera to rotate to the direction of shooting the image of the mouse click position according to the PT value of the gun binding pan-tilt camera.

4. A server of a gun-ball linkage, characterized by, For executing the following steps: The curtain binding gun information and the pixel coordinates of the mouse click position sent by the three-dimensional engine are received; Based on the curtain binding gun information and the pixel coordinates of the mouse click position, the PT value of the gun binding pan-tilt camera is calculated through the homography matrix configuration file; The PT value of the gun binding pan-tilt camera is returned to the three-dimensional engine.

5. The server of claim 4, wherein, The PT value of the gun binding pan-tilt camera is calculated based on the curtain binding gun information and the pixel coordinates of the mouse click position through the homography matrix configuration file, including the following steps: The homography matrix configuration file is loaded; wherein the homography matrix configuration file includes the homography matrix of the gun and the pan-tilt camera, and the configuration information of the gun binding pan-tilt camera; According to the curtain binding gun information and the pixel coordinates of the mouse click position, the homography matrix of the gun and the pan-tilt camera, and the configuration information of the gun binding pan-tilt camera in the homography matrix configuration file are obtained, and the PT value of the gun binding pan-tilt camera is calculated.

6. The server of claim 5, wherein, The homography matrix configuration file includes: the homography matrix and configuration information of one gun binding one pan-tilt camera, and the homography matrix and configuration information of one gun binding two or more pan-tilt cameras.

7. A server of the gun-ball linkage according to claim 5 or 6, wherein, The homography matrix of the gun and the pan-tilt camera in the homography matrix configuration file is obtained, including the following steps: The homography matrix of the gun and the pan-tilt camera in the homography matrix configuration file is obtained by calling the findHomography function.

8. A gun-ball linkage system characterized by, A three-dimensional engine of gun-pan-tilt camera linkage according to any one of claims 1 to 3, and a server of gun-pan-tilt camera linkage according to any one of claims 4 to 7 connected with the three-dimensional engine of gun-pan-tilt camera linkage.

9. A gun-ball linkage system according to claim 8, wherein, Also include a server connection calibration tool end linked with the gun ball, for executing the following steps: According to the gun pixel coordinates and ball matching pair calculation gun and ball homography matrix; Gun and ball homography matrix, gun binding ball machine configuration information is saved as a homography matrix configuration file; Send the homography matrix configuration file to the server; Wherein, the generation method of gun pixel coordinates and ball matching pair includes one or two of the following: Manual calibration: manual calibration generates gun pixel coordinates and ball PT value matching pair; Automatic calibration, including the following steps: The ball machine automatically scans the image and saves the scanned image; Wherein, each image corresponds to a PT value; The ball machine saves the image and the gun machine saves the image feature matching, according to the set threshold value to judge whether the matching is successful; If the matching is successful, the gun pixel coordinates of the ball machine saved image matching successful are calculated; The matching successful ball machine information, gun pixel coordinates as a gun pixel coordinates and ball PT value matching pair.

10. A gunball linkage method, characterized by, Use a gun ball linkage system of claim 8 or 9 to carry out gun ball linkage, including the following steps: When the mouse clicks the three-dimensional scene, the three-dimensional engine acquires the mouse click action, and judges whether the mouse click position falls in the curtain: if yes, go to the next step; If not, do not act; Three-dimensional engine gets the pixel coordinates of the mouse click position in the curtain binding image; Wherein, the curtain binding image refers to the image taken by the gun machine bound to the curtain; Three-dimensional engine sends the curtain binding gun machine information and the pixel coordinates of the mouse click position to the server; The server receives the curtain binding gun machine information and the pixel coordinates of the mouse click position sent by the three-dimensional engine; The server calculates the PT value of the gun binding ball machine based on the curtain binding gun machine information and the pixel coordinates of the mouse click position through the homography matrix configuration file; The server returns the PT value of the gun binding ball machine to the three-dimensional engine; Three-dimensional engine receives the PT value of the gun binding ball machine; Wherein, the PT value refers to the horizontal rotation value and the vertical rotation value; Three-dimensional engine calls the ball machine to rotate to the direction of shooting the image of the mouse click position according to the PT value of the gun binding ball machine.

Images