A gun-ball linkage method, calibration method and device

Abstract

The application discloses a gun-ball linkage method, a calibration method and device, relates to the technical field of video monitoring and the technical field of thermal imaging, and can realize automatic calibration between a gun and a ball camera. The method comprises the following steps: acquiring a pixel coordinate of a target object in a target thermal imaging image in a target thermal imaging image collected by a thermal imaging image collection device; converting the pixel coordinate into a PTZ coordinate of the target object according to a target conversion relationship; the target conversion relationship is a coordinate conversion relationship between the pixel coordinate in the thermal imaging image collected by the thermal imaging image collection device and the PTZ coordinate of the ball camera; and adjusting the ball camera to a specified state according to the PTZ coordinate of the target object; wherein the specified state is a state of the ball camera when the target object is located at a preset position in a field of view range of the ball camera.

Description

Technical Field

[0001] This application relates to the fields of video surveillance technology and thermal imaging technology, and in particular to a method, calibration method and device for camera-ball linkage. Background Technology

[0002] Currently, with the rapid development of the surveillance field, standalone bullet cameras (or simply bullet cameras) or standalone dome cameras (or simply dome cameras) can no longer meet the increasingly complex surveillance scenarios. Bullet cameras have a wide field of view and a broad monitoring range; while dome cameras have a smaller field of view and a narrower monitoring range, they can better monitor details by controlling PTZ information (movement up, down, left, right, zoom, and magnification). The combined use of bullet and dome cameras is becoming increasingly common, better meeting the needs of increasingly complex surveillance scenarios.

[0003] When bullet cameras and PTZ cameras are used in combination, they are typically controlled by electronic equipment. During this control, the user needs to manually select points or frames within the bullet camera's field of view to calibrate the two cameras (i.e., establish the correspondence between the image captured by the bullet camera and the PTZ coordinates of the PTZ camera). Therefore, existing calibration methods require manual labor, which is time-consuming, inefficient, and susceptible to subjective human error, affecting calibration accuracy. Summary of the Invention

[0004] This application provides a gun-ball linkage method, calibration method, and device that can realize automatic calibration between the gun and the ball camera, thereby helping to improve the accuracy of calibration.

[0005] In a first aspect, this application provides a method for gun-to-ball linkage, applied to a gun-to-ball linkage system, the gun-to-ball linkage system comprising: a first gun camera and a PTZ camera, the first gun camera including a thermal imaging image acquisition device; the method comprising: acquiring the pixel coordinates of a target object in a target thermal imaging image acquired by the thermal imaging image acquisition device; converting the pixel coordinates into the PTZ coordinates of the target object according to a target transformation relationship; the target transformation relationship being the coordinate transformation relationship between the pixel coordinates in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera; adjusting the PTZ camera to a specified state according to the PTZ coordinates of the target object; wherein, the specified state is the state of the PTZ camera when the target object is located at a preset position within the field of view of the PTZ camera.

[0006] Based on the technical solution provided in this application, at least the following beneficial effects can be achieved: Based on the coordinate transformation relationship between the pixel coordinates in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera, after the thermal imaging image acquisition device outputs the position of the target object in the thermal imaging image, the PTZ coordinates of the PTZ camera can be directly obtained according to this target transformation relationship. The PTZ camera can then be adjusted according to these PTZ coordinates, achieving camera-PTZ linkage. In this way, throughout the entire process of camera-PTZ linkage, based on this target transformation relationship, automatic calibration between the pixel coordinates in the thermal imaging image acquired by the camera's thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera can be achieved, improving the calibration efficiency between the camera and the PTZ camera. Furthermore, since the imaging features of thermal imaging are unclear, manual calibration is prone to deviation. Therefore, the technical solution provided in this application eliminates the need for manual calibration by the user, avoiding the influence of subjective human factors and improving the accuracy of calibration between the camera and the PTZ camera.

[0007] Optionally, the PTZ linkage system includes: at least one camera, which includes a thermal imaging image acquisition device and a visible light image acquisition device; the method further includes: determining a first transformation relationship; wherein the first transformation relationship is a coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device; determining a second transformation relationship; wherein the second transformation relationship is a coordinate transformation relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera; and determining a target transformation relationship based on the first transformation relationship and the second transformation relationship.

[0008] Optionally, the angle between the optical axis of the thermal imaging lens in the thermal imaging image acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is less than a preset threshold.

[0009] Optionally, the aforementioned first conversion relationship is characterized by a scaling factor and an offset factor between the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device; the scaling factor is the ratio of the width to the height of the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device; the offset factor is the coordinate difference between the coordinates of a pixel in the thermal image acquired by the thermal imaging device and the coordinates of the corresponding pixel in the visible light image acquired by the visible light image acquisition device; wherein, the pixels in the thermal image acquired by the thermal imaging device and the corresponding pixels in the visible light image acquired by the visible light image acquisition device represent pixels of the same object, or pixels representing the same part of the same object;

[0010] Optionally, determining the second transformation relationship includes: determining the second transformation relationship based on the transformation relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the world coordinates, and the transformation relationship between the world coordinates and the PTZ coordinates of the PTZ camera.

[0011] Optionally, determining the target transformation relationship based on the first transformation relationship and the second transformation relationship includes: representing the coordinates of pixels in the visible light image acquired by the visible light image acquisition device using the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship; and replacing the coordinates of pixels in the visible light image acquired by the visible light image acquisition device in the second transformation relationship with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship to obtain the target transformation relationship.

[0012] Optionally, the aforementioned first gun mechanism includes a visible light image acquisition device;

[0013] Alternatively, the aforementioned at least one firing mechanism may further include a second firing mechanism, which includes a visible light image acquisition device.

[0014] Optionally, the above method further includes receiving a target transformation relationship. That is, embodiments of this application support receiving target transformation relationships from other devices.

[0015] Secondly, this application provides a calibration method applied to a PTZ camera linkage system, comprising at least one camera and one PTZ camera, wherein the at least one camera includes a thermal imaging image acquisition device and a visible light image acquisition device; the method comprises: determining a first transformation relationship; wherein the first transformation relationship is a coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device; determining a second transformation relationship; wherein the second transformation relationship is a coordinate transformation relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera; and determining a target transformation relationship based on the first and second transformation relationships, wherein the target transformation relationship is a coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera.

[0016] Based on the technical solution provided in this application, at least the following beneficial effects can be achieved: According to the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device, and the coordinate transformation relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera, the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera is obtained, realizing automatic calibration between the bullet camera and the PTZ camera, eliminating the need for manual operation by the user, reducing manpower consumption, improving calibration efficiency, and avoiding the influence of subjective human factors, thus improving the accuracy of calibration.

[0017] Optionally, the angle between the optical axis of the thermal imaging lens in the thermal imaging image acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is less than a preset threshold.

[0018] Optionally, determining the target transformation relationship based on the first transformation relationship and the second transformation relationship includes: representing the coordinates of pixels in the visible light image acquired by the visible light image acquisition device using the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship; and replacing the coordinates of pixels in the visible light image acquired by the visible light image acquisition device in the second transformation relationship with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship to obtain the target transformation relationship.

[0019] Optionally, the at least one gun bolt includes: a first gun bolt, the first gun bolt including a thermal imaging image acquisition device and a visible light image acquisition device; or, the at least one gun bolt includes: a first gun bolt and a second gun bolt; wherein, the first gun bolt includes a thermal imaging image acquisition device; and the second gun bolt includes a visible light image acquisition device.

[0020] Thirdly, this application provides a PTZ (Pocket Camera and Telescope Camera) linkage device, applied to a PTZ linkage system. The PTZ linkage system includes: a first PTZ camera and a telescope camera. The first PTZ camera includes a thermal imaging image acquisition device. The device includes: an acquisition module for acquiring the pixel coordinates of a target object in the target thermal imaging image acquired by the thermal imaging image acquisition device; a conversion module for converting the pixel coordinates into the PTZ coordinates of the target object according to a target conversion relationship; the target conversion relationship is the coordinate conversion relationship between the pixel coordinates in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the telescope camera; and a linkage module for adjusting the telescope camera to a specified state according to the PTZ coordinates of the target object; wherein, the specified state is the state of the telescope camera when the target object is located at a preset position within the field of view of the telescope camera.

[0021] Optionally, the PTZ linkage system includes: at least one camera, which includes a thermal imaging image acquisition device and a visible light image acquisition device; the aforementioned PTZ linkage device further includes: a calibration module; the calibration module is used to determine a first conversion relationship; wherein the first conversion relationship is a coordinate conversion relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device; determine a second conversion relationship; wherein the second conversion relationship is a coordinate conversion relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera; and determine a target conversion relationship based on the first conversion relationship and the second conversion relationship.

[0022] Optionally, the angle between the optical axis of the thermal imaging lens in the thermal imaging image acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is less than a preset threshold.

[0023] Optionally, the aforementioned first conversion relationship is characterized by a scaling factor and an offset factor between the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device; the scaling factor is the ratio of the width of the thermal image acquired by the thermal imaging device to the ratio of the height of the visible light image acquired by the visible light image acquisition device; the offset factor is the coordinate difference between the coordinates of a pixel in the thermal image acquired by the thermal imaging device and the coordinates of the corresponding pixel in the visible light image acquired by the visible light image acquisition device; wherein, the pixels in the thermal image acquired by the thermal imaging device and the corresponding pixels in the visible light image acquired by the visible light image acquisition device represent pixels of the same object, or pixels representing the same part of the same object.

[0024] Optionally, the calibration module described above is specifically used to determine the second conversion relationship based on the conversion relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the world coordinates, and the conversion relationship between the world coordinates and the PTZ coordinates of the PTZ camera.

[0025] Optionally, the calibration module is specifically used to represent the coordinates of pixels in the visible light image acquired by the visible light image acquisition device with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and a first transformation relationship; and to replace the coordinates of pixels in the visible light image acquired by the visible light image acquisition device in the second transformation relationship with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship to obtain the target transformation relationship.

[0026] Optionally, the first gun bolt includes a visible light image acquisition device; or, at least one gun bolt further includes a second gun bolt, which includes a visible light image acquisition device.

[0027] Optionally, the acquisition module described above is also used to receive the target transformation relationship.

[0028] Fourthly, this application provides a calibration device applied to a PTZ camera linkage system, the PTZ camera linkage system comprising: at least one camera and one PTZ camera, the at least one camera comprising a thermal imaging image acquisition device and a visible light image acquisition device; the device comprising: a calibration module for determining a first conversion relationship; wherein the first conversion relationship is a coordinate conversion relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device; determining a second conversion relationship; wherein the second conversion relationship is a coordinate conversion relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera; and determining a target conversion relationship based on the first conversion relationship and the second conversion relationship, the target conversion relationship being a coordinate conversion relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera.

[0029] Optionally, the angle between the optical axis of the thermal imaging lens in the thermal imaging image acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is less than a preset threshold.

[0030] Optionally, the calibration module is specifically used to represent the coordinates of pixels in the visible light image acquired by the visible light image acquisition device with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and a first transformation relationship; and to replace the coordinates of pixels in the visible light image acquired by the visible light image acquisition device in the second transformation relationship with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship to obtain the target transformation relationship.

[0031] Optionally, the at least one gun bolt includes: a first gun bolt, the first gun bolt including a thermal imaging image acquisition device and a visible light image acquisition device; or, the at least one gun bolt includes: a first gun bolt and a second gun bolt; wherein, the first gun bolt includes a thermal imaging image acquisition device; and the second gun bolt includes a visible light image acquisition device.

[0032] Fifthly, this application provides a ball-and-gun linkage device, comprising: one or more processors; one or more memory; wherein the one or more memory is used to store computer program code, the computer program code including computer instructions, and when the one or more processors execute the computer instructions, the ball-and-gun linkage device executes any of the ball-and-gun linkage methods provided in the first aspect above.

[0033] In a sixth aspect, this application provides a calibration apparatus, comprising: one or more processors; one or more memories; wherein the one or more memories are used to store computer program code, the computer program code including computer instructions, and when the one or more processors execute the computer instructions, the calibration apparatus performs any of the calibration methods provided in the second aspect above.

[0034] In a seventh aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed on a computer, cause the computer to perform any of the methods provided in the first and / or second aspects described above.

[0035] The beneficial effects of aspects two through seven mentioned above can be referred to in aspect one or two, and will not be repeated here. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a gun-ball linkage system provided in an embodiment of this application;

[0037] Figure 2 A flowchart illustrating a gun-ball linkage method provided in this application embodiment;

[0038] Figure 3 A schematic diagram of a target thermal imaging image provided in an embodiment of this application;

[0039] Figure 4 A flowchart illustrating a calibration method provided in an embodiment of this application;

[0040] Figure 5 A flowchart illustrating another calibration method provided in this application embodiment;

[0041] Figure 6 A calibration diagram of a bolt carrier provided for an embodiment of this application;

[0042] Figure 7 A calibration diagram of another bolt action provided in an embodiment of this application;

[0043] Figure 8 A flowchart illustrating yet another calibration method provided in this application embodiment;

[0044] Figure 9 A schematic diagram of coordinate system transformation provided in an embodiment of this application;

[0045] Figure 10 This is a schematic diagram of the structure of a gun-ball linkage device provided in an embodiment of this application;

[0046] Figure 11 This is a schematic diagram of the structure of a calibration device provided in an embodiment of this application;

[0047] Figure 12 This is a schematic diagram of a gun-ball linkage device provided in an embodiment of this application. Detailed Implementation

[0048] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0049] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0050] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0051] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0052] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0053] As described in the background section, in existing technologies, when bullet cameras and PTZ cameras are used in combination, electronic devices typically control their linkage. During this linkage control, the user needs to manually select points or frames within the bullet camera's field of view to calibrate the two cameras (i.e., establish the correspondence between pixel coordinates in the bullet camera's image and the PTZ coordinates of the PTZ camera). Therefore, the calibration methods provided by existing technologies require manual labor, which is time-consuming, inefficient, and susceptible to subjective human factors, affecting calibration accuracy.

[0054] To address the aforementioned technical problems, this application provides an automatic calibration method for a PTZ camera linkage system. The method is based on the coordinate transformation relationship between the pixel coordinates in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera. After the thermal imaging lens outputs the position of the target object in the thermal imaging image, the PTZ coordinates of the PTZ camera can be directly obtained based on this target transformation relationship. The PTZ camera then adjusts according to these PTZ coordinates, achieving PTZ camera linkage. In this way, throughout the entire process of PTZ camera linkage, automatic calibration between the pixel coordinates in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera can be achieved based on this target transformation relationship, improving the calibration efficiency between the PTZ camera and the camera. Furthermore, since the imaging features of thermal imaging are unclear, manual calibration is prone to errors. Therefore, the technical solution provided in this application eliminates the need for manual calibration by the user, avoiding the influence of subjective human factors and improving the accuracy of calibration between the PTZ camera and the camera.

[0055] like Figure 1 The diagram shown is a schematic of a PTZ (Pad-to-Patrol) linkage system provided in an embodiment of this application. The PTZ linkage system includes a camera 10, a PTZ camera 20, and a controller 30. The camera 10 and the PTZ camera 20 can be connected via wired or wireless means, as can the camera 10 and the controller 30, and the PTZ camera 20 and the controller 30.

[0056] The bullet camera 10 is a surveillance camera with a rectangular, cylindrical, or various near-cylindrical shape. The bullet camera 10 includes a C / CS lens interface.

[0057] In some embodiments, the receiver 10 includes an image acquisition device, which includes at least one of the following: a lens, an imaging sensor, a chip processor, etc. For example, the image acquisition device includes a thermal imaging image acquisition device; the thermal imaging image acquisition device includes at least one of the following: a thermal imaging lens, a thermal imaging sensor, a chip processor, etc. Yet another example, the image acquisition device includes a visible light image acquisition device; the visible light image acquisition device includes at least one of the following: a visible light lens, a visible light imaging sensor, a chip processor, etc.

[0058] Thermal imaging lenses, employing thermal infrared imaging technology, can detect extremely small temperature differences and convert them into real-time video images for display. However, the video images captured by thermal imaging lenses only show the thermal outlines of people and objects, not their true appearance. Therefore, this embodiment uses a thermal imaging lens on the gun receiver 10 to lock onto the target.

[0059] Visible light lenses have a strong mapping effect on electromagnetic waves with wavelengths between 400nm and 760nm (the portion of the electromagnetic spectrum that the human eye can perceive). Video images captured by visible light lenses can clearly show the true appearance of objects. In this embodiment, the visible light lens of the gun 10 is used to monitor the overall picture.

[0060] In some embodiments, the field of view of a thermal imaging lens is smaller than that of a visible light lens.

[0061] Optionally, the bolt carrier 10 is a binocular bolt carrier, including a thermal imaging image acquisition device and a visible light image acquisition device; or, the bolt carrier 10 includes a first bolt carrier and a second bolt carrier; wherein the first bolt carrier includes a thermal imaging image acquisition device; and the second bolt carrier includes a visible light image acquisition device. The angle between the optical axis of the thermal imaging lens in the thermal imaging acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is less than a preset threshold, for example, 0.1 degrees. When the angle between the optical axis of the thermal imaging lens in the thermal imaging acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is 0 degrees, the optical axis of the thermal imaging lens in the thermal imaging acquisition device is parallel to the optical axis of the visible light lens in the visible light image acquisition device.

[0062] The PTZ camera 20, also known as a dome camera, includes an integrated camera (with a zoom lens) and a pan-tilt structure. The PTZ camera 20 can rotate up, down, left, and right under the control of a control terminal to achieve a monitoring effect.

[0063] In some embodiments, the PTZ camera 20 includes a visible light image acquisition device (including a visible light lens) capable of reflecting electromagnetic waves with wavelengths between 400nm and 760nm to capture clear video images.

[0064] In some embodiments, the aforementioned PTZ camera 20 is a PTZ camera. The PTZ parameters of the PTZ camera are: horizontal angle P, vertical angle T, and zoom parameter Z. Three-dimensional PTZ coordinates are obtained from the PTZ parameters; the three-dimensional PTZ coordinates represent the center point position of the image captured by the PTZ camera. The PTZ camera has a preset position where the horizontal angle P and vertical angle T are both 0. When the PTZ camera rotates right from the preset position, the horizontal angle P increases sequentially until it reaches its maximum value and then becomes 0. When the PTZ camera rotates upward from the preset position, the vertical angle T increases; when the PTZ camera rotates downward from the preset position, the vertical angle T decreases. The larger the PTZ camera zoom parameter Z value, the greater the magnification of the center point of the image. Therefore, any point in the PTZ camera corresponds to a value of horizontal angle P, vertical angle T, and zoom parameter Z, and this point corresponds to a pixel (x, y) in the camera's image, with a one-to-one correspondence.

[0065] The controller 30 is used to link the bullet camera 10 and the PTZ camera 20, such as controlling the PTZ camera 20 to move by using information from the image of the target object acquired by the bullet camera 10.

[0066] Optionally, the controller 30 can be a standalone device; or, the controller 30 can be integrated with the camera module 10; or, the controller 30 can be integrated with the PTZ camera 20. The embodiments provided in this application... Figure 1 The example provided uses the independent configuration of the camera 10, the PTZ camera 20, and the controller 30.

[0067] In some embodiments, a pan-tilt unit is provided under the PTZ camera 20 to support the PTZ camera, and the controller 30 controls the movement of the PTZ camera 20 by controlling the up, down, left and right movement of the pan-tilt unit.

[0068] The embodiments provided in this application will now be described in detail with reference to the accompanying drawings.

[0069] The gun-ball linkage method provided in this application embodiment is applied to a gun-ball linkage system, which includes a first gun camera and a ball camera, wherein the first gun camera includes a thermal imaging image acquisition device. Figure 1 As shown, the ball-and-gun linkage system also includes a controller, which is used to execute the ball-and-gun linkage method provided in the embodiments of this application. Figure 2 As shown, the method includes the following steps:

[0070] S101. Obtain the pixel coordinates of the target object in the target thermal imaging image acquired by the thermal imaging image acquisition device.

[0071] Among them, the target thermal imaging image is the thermal imaging image output by the thermal imaging image acquisition device when it detects the target object.

[0072] The target object is a moving target (i.e., a dynamic target) within the field of view of the thermal imaging image acquisition device. For example, the target object can be a person or an animal.

[0073] In some embodiments, when a target object appears within the field of view of a thermal imaging image acquisition device, the thermal imaging image acquisition device can generate a target thermal imaging image based on the temperature difference (or infrared difference) between the target object and other objects within the field of view, and identify the target object based on the target thermal imaging image, generating a thermal imaging image such as... Figure 3 The target bounding box of the target object shown.

[0074] Optionally, the pixel coordinates of the target object in the target thermal image can be the coordinates of a preset position of the target object's bounding box (such as the top left corner, top right corner, bottom left corner, bottom right corner, or center point), as well as the width and height of the bounding box; or, the position coordinates of the target object in the target thermal image can be the coordinates of the top left corner and bottom right corner (or, the coordinates of the bottom left corner and top right corner) of the target object's bounding box. This application does not specifically limit the form of the position coordinates of the target object in the target thermal image, as long as it clearly represents the position of the target object in the target thermal image.

[0075] For example, such as Figure 3 As shown, the pixel coordinates of the target object in the target thermal imaging image can be the coordinates (x, y) of the upper left corner of the target object's bounding box and its width and height (w, h).

[0076] S102. Based on the target transformation relationship, convert the pixel coordinates into the PTZ coordinates of the target object.

[0077] The target transformation relationship refers to the coordinate transformation relationship between the pixel coordinates in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera. Optionally, the target transformation relationship can be determined by the controller itself; or, the target transformation relationship can be generated by other devices and sent to the controller of the PTZ camera system. The other devices can be any other devices besides the device where the controller is located, and this application does not limit this.

[0078] PTZ is an abbreviation for Pan / Tilt / Zoom, representing omnidirectional (left / right / up / down) movement and lens zoom / magnification control.

[0079] In some embodiments, the above-mentioned gun-ball linkage system includes: at least one gun; the at least one gun includes the above-mentioned thermal imaging image acquisition device and visible light image acquisition device.

[0080] In some embodiments, the angle between the optical axis of the thermal imaging lens in the thermal imaging acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is less than a preset threshold, or, for example, the preset threshold may be 0.1 degrees. When the angle between the optical axis of the thermal imaging lens in the thermal imaging acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is 0 degrees, the optical axes of the thermal imaging lens in the thermal imaging acquisition device and the optical axes of the visible light lens in the visible light image acquisition device are parallel.

[0081] Optionally, the first bolt carrier may further include a visible light image acquisition device; or, at least one bolt carrier may include a second bolt carrier, which may include a visible light image acquisition device.

[0082] In some embodiments, the aforementioned target transformation relationship can be determined based on a first transformation relationship and a second transformation relationship. The first transformation relationship is the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device; the second transformation relationship is the coordinate transformation relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera.

[0083] The first transformation relationship is characterized by the scaling factor and offset factor between the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device. The scaling factor is the ratio of the width to the height of the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device; the offset factor is the coordinate difference between the coordinates of a pixel in the thermal image acquired by the thermal imaging device and the coordinates of the corresponding pixel in the visible light image acquired by the visible light image acquisition device.

[0084] In this context, pixels in thermal images acquired by thermal imaging acquisition devices and corresponding pixels in visible light images acquired by visible light acquisition devices represent the same object, or pixels representing the same part of the same object.

[0085] It is understandable that, since the first transformation relationship is characterized by scaling and offset factors, when the optical axis of the thermal imaging lens in the thermal imaging image acquisition device is parallel or approximately parallel to the optical axis of the visible light lens in the visible light image acquisition device, the pixel coordinates of the target object in the thermal imaging image obtained by the first transformation relationship based on the pixel coordinates of the target object in the visible light image have a small error compared with the actual pixel coordinates of the target object in the thermal imaging image. This can improve the accuracy of calibration between the thermal imaging image and the visible light image, thereby helping to improve the accuracy of calibration between the bullet camera and the PTZ camera.

[0086] The second conversion relationship is determined based on the conversion relationship between the pixel coordinates and world coordinates in the visible light image acquired by the visible light image acquisition device, and the conversion relationship between the world coordinates and the PTZ coordinates of the PTZ camera.

[0087] As one possible implementation, the coordinates of pixels in the visible light image acquired by the visible light image acquisition device are represented by the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and a first transformation relationship; then, the coordinates of pixels in the visible light image acquired by the visible light image acquisition device in the second transformation relationship are replaced with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship, thereby obtaining the target transformation relationship.

[0088] S103. Adjust the PTZ coordinates of the target object to the specified state.

[0089] The specified state refers to the state of the PTZ camera when the target object is located at a preset position within the PTZ camera's field of view.

[0090] The preset position is the location within the PTZ camera's field of view where the visual effect achieves a preset visual effect. For example, the preset position can be the center position within the PTZ camera's field of view; or, the preset position can be any position within a circular area constructed with the center position within the PTZ camera's field of view as the center and a specific length as the radius, etc.

[0091] Based on the technical solution provided in this application, at least the following beneficial effects can be achieved: Based on the coordinate transformation relationship between the pixel coordinates in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera, after the thermal imaging image acquisition device outputs the position of the target object in the thermal imaging image, the PTZ coordinates of the PTZ camera can be directly obtained according to this target transformation relationship. The PTZ camera can then be adjusted according to these PTZ coordinates, achieving camera-PTZ linkage. In this way, throughout the entire process of camera-PTZ linkage, based on this target transformation relationship, automatic calibration between the pixel coordinates in the thermal imaging image acquired by the camera's thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera can be achieved, improving the calibration efficiency between the camera and the PTZ camera. Furthermore, since the imaging features of thermal imaging are unclear, manual calibration is prone to deviation. Therefore, the technical solution provided in this application eliminates the need for manual calibration by the user, avoiding the influence of subjective human factors and improving the accuracy of calibration between the camera and the PTZ camera.

[0092] This application provides a calibration method for a PTZ camera system. The PTZ camera system includes at least one camera and one PTZ camera, and the at least one camera includes a thermal imaging image acquisition device and a visible light image acquisition device.

[0093] Wherein, the angle between the optical axis of the thermal imaging lens in the thermal imaging acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is less than a preset threshold, or, for example, the preset threshold can be 0.1 degrees. When the angle between the optical axis of the thermal imaging lens in the thermal imaging acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is 0 degrees, the optical axes of the thermal imaging lens in the thermal imaging acquisition device and the optical axes of the visible light lens in the visible light image acquisition device are parallel.

[0094] Optionally, the at least one gun bolt includes: a first gun bolt, the first gun bolt including a thermal imaging image acquisition device and a visible light image acquisition device; or, the at least one gun bolt includes: a first gun bolt and a second gun bolt; wherein, the first gun bolt includes a thermal imaging image acquisition device; and the second gun bolt includes a visible light image acquisition device.

[0095] It is understood that, in the embodiments of this application, "thermal imaging image acquired by thermal imaging image acquisition device" refers to the thermal imaging image acquired by the thermal imaging image acquisition device of the bullet camera; "visible light image acquired by visible light image acquisition device" refers to the visible light image acquired by the visible light image acquisition device of the bullet camera. In the embodiments of this application, "visible light image of PTZ camera" refers to the visible light image acquired by the visible light image acquisition device of the PTZ camera.

[0096] like Figure 4 As shown, the method also includes the following steps:

[0097] S201. Determine the first transformation relationship.

[0098] The first transformation relationship is the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device.

[0099] Optional, such as Figure 5 As shown, step S201 can be specifically implemented as follows:

[0100] S2011. Determine the scaling factor between the thermal imaging image acquired by the thermal imaging image acquisition device and the visible light image acquired by the visible light image acquisition device.

[0101] The scaling factor is the ratio of the width of the thermal image acquired by the thermal imaging image acquisition device to the height of the visible light image acquired by the visible light image acquisition device.

[0102] In some embodiments, the scaling factor is determined based on the ratio of the width of the thermal image acquired by the thermal imaging image acquisition device to the ratio of the height of the visible light image acquired by the visible light image acquisition device.

[0103] Specifically, four heated markers at known locations are placed within the field of view of the bolt carrier. These markers clearly define the boundaries of at least one of the bolt carrier's shooting ranges. For example... Figure 6 As shown, based on the four markers, the imaging range of the thermal imaging image acquisition device and the imaging range of the visible light image acquisition device can be clearly defined. That is, the position and size of the thermal imaging image acquired by the thermal imaging image acquisition device, as well as the position and size of the visible light image acquired by the visible light image acquisition device, can be clearly determined.

[0104] For example, such as Figure 7As shown, assuming the height of the thermal image acquired by the thermal imaging image acquisition device is H1 and the width is W1, and the height of the visible light image acquired by the visible light image acquisition device is H2 and the width is W2, then the scaling factor (scale) between the thermal image acquired by the thermal imaging image acquisition device and the visible light image acquired by the visible light image acquisition device in the vertical direction is... y =H2 / H1, the scaling factor in the horizontal direction. x =W2 / W1.

[0105] S2012. Determine the offset coefficient between the thermal imaging image acquired by the thermal imaging image acquisition device and the visible light image acquired by the visible light image acquisition device.

[0106] The offset coefficient is the coordinate difference between the coordinates of a pixel in the thermal image acquired by the thermal imaging image acquisition device and the coordinates of the corresponding pixel in the visible light image acquired by the visible light image acquisition device.

[0107] In this context, pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and corresponding pixels in the visible light image acquired by the visible light image acquisition device represent the same object, or pixels representing the same part of the same object. For example, the object can be a person or an animal.

[0108] Optionally, the pixels in the thermal imaging image acquired by the thermal imaging image acquisition device can be pixels at any location in the thermal imaging image. For example, the pixel could be the pixel at the top left corner of the thermal imaging image acquired by the thermal imaging image acquisition device.

[0109] In some embodiments, an offset coefficient is determined based on the coordinate difference between the coordinates of a pixel in a thermal imaging image acquired by a thermal imaging image acquisition device and the coordinates of the corresponding pixel in a visible light image acquired by a visible light image acquisition device.

[0110] For example, such as Figure 7 As shown, assuming the coordinates of the top-left pixel in the thermal image acquired by the thermal imaging device are (x1, y1), and the coordinates of the top-left pixel in the visible light image acquired by the visible light image acquisition device are (x2, y2), then the offset coefficient between the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device in the X-axis direction is offset. x =x1-scale x *x2; The offset coefficient in the Y-axis direction is offset. y =y1-scale y *y2.

[0111] S2013. Determine the first conversion relationship based on the scaling factor and offset factor between the thermal imaging image acquired by the thermal imaging image acquisition device and the visible light image acquired by the visible light image acquisition device.

[0112] In some embodiments, the first conversion relationship constructed based on the scaling factor and offset factor between the thermal imaging image acquired by the thermal imaging image acquisition device and the visible light image acquired by the visible light image acquisition device satisfies the following formula (1):

[0113]

[0114] Among them, (x vis y vis (x) represents the coordinates of a pixel in a visible light image acquired by the visible light image acquisition device. ifr y ifr ) represents the coordinates of pixels in a thermal imaging image acquired by a thermal imaging image acquisition device.

[0115] Thus, by establishing the first transformation relationship, coordinate transformation can be achieved between the coordinates of pixels in the thermal image acquired by the thermal imaging device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device. This eliminates the need for users to manually select feature points or target boxes, enabling automatic calibration of the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device, reducing manual labor and improving conversion efficiency.

[0116] Furthermore, since the first conversion relationship is characterized by scaling and offset factors, this embodiment sets the optical axis of the thermal imaging lens in the thermal imaging image acquisition device to be parallel or approximately parallel to the optical axis of the visible light lens in the visible light image acquisition device. This improves the accuracy of calibration between the thermal imaging image and the visible light image, and also expands the applicability of the first conversion relationship. (It is understood that if there is an angle between the optical axis of the thermal imaging lens and the optical axis of the visible light lens, the farther the target object is from the lens, the greater the difference between the position of the target object in the thermal imaging image and the position of the target object in the visible light image. Therefore, when the optical axis of the thermal imaging lens is parallel or approximately parallel to the optical axis of the visible light lens, the distance factor has less impact on the accuracy of calibration between the thermal imaging image and the visible light image.)

[0117] S202, Determine the second transformation relationship.

[0118] The second transformation relationship is the coordinate transformation relationship between the pixel coordinates in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera.

[0119] Optional, such as Figure 8As shown, step S202 can be specifically implemented as follows:

[0120] S2021. Construct the transformation relationship between pixel coordinates and world coordinates in the visible light image acquired by the visible light image acquisition device.

[0121] Specifically, the transformation relationship between the pixel coordinate system and the world coordinate system of the visible light image acquired by the visible light image acquisition device is constructed.

[0122] In some embodiments, the visible light image acquisition device of at least one gun camera is calibrated to obtain the camera's intrinsic parameter matrix and extrinsic parameter matrix. The camera's intrinsic parameter matrix satisfies the following formula (2), and the camera's extrinsic parameter matrix satisfies the following formula (3):

[0123]

[0124] Where f is the focal length, dX is the physical length of the pixel on the camera sensor in the X direction, dY is the physical length of the pixel on the camera sensor in the Y direction, and (u0, v0) are the coordinates of the sensor center in the pixel coordinate system.

[0125]

[0126] Where R is the rotation matrix and T is the rotation vector.

[0127] Based on the above formulas (2) and (3), the transformation relationship between the pixel coordinate system and the world coordinate system of the visible light image acquired by the visible light image acquisition device is constructed. This transformation relationship satisfies the following formula (4):

[0128]

[0129] Where (U, V, W) are the physical coordinates of a point in the world coordinate system, (x, y) are the pixel coordinates of that point in the pixel coordinate system, and Z is the scale factor.

[0130] Let H be the product of the camera's intrinsic and extrinsic parameter matrices. H is the homography matrix, used to reflect the transformation relationship between pixel coordinates and world coordinates. Define W = 0, from which we can obtain the following formula (5):

[0131]

[0132] Furthermore, by eliminating the scale factor Z, the constraint equation between the pixel coordinates and world coordinates in the visible light image acquired by the visible light image acquisition device can be obtained. This constraint equation satisfies the following formula (6):

[0133]

[0134] In formula (6), the homography matrix H is a homogeneous matrix. 33 If the result is normalized to 1, the remaining 8 independent elements can be obtained by least squares regression. When solving for matrix H, four non-collinear points (e.g., the four points corresponding to a 2x2 grid) can be used to obtain the corresponding matrix H. If there are more than four points, the optimal matrix H can be obtained by least squares regression.

[0135] S2022. Establish the conversion relationship between world coordinates and PTZ coordinates of the PTZ camera.

[0136] In some embodiments, the elevation tilt angle Pan and horizontal angle Tilt in the PTZ coordinates of the PTZ camera can be reflected in the world coordinate system as follows: Figure 9 The form shown, where, Figure 9 In the diagram, point O is the location of the PT camera, point A is the location of the target, and h is the installation height of the PT camera. For example, ignoring the target height, the transformation relationship between world coordinates (U,V) and the PT camera's PT(Pan,Tilt) coordinates can satisfy the following formula (7):

[0137]

[0138] S2023. Based on the conversion relationship between the pixel coordinates in the visible light image acquired by the visible light image acquisition device and the world coordinates, and the conversion relationship between the world coordinates and the PTZ coordinates of the PTZ camera, a second conversion relationship is obtained (i.e., the conversion relationship between the pixel coordinates in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera).

[0139] In some embodiments, by combining the above formulas (5) and (7), the PT coordinates of the PT camera can be transformed to the coordinates of pixels in the visible light image acquired by the visible light image acquisition device. This transformation relationship satisfies the following formula (8):

[0140]

[0141] Furthermore, according to the above formula (8), the transformation relationship between the pixel coordinates in the visible light image acquired by the visible light image acquisition device and the PT coordinate system of the PT camera can be obtained. That is, the pixel coordinates in the visible light image acquired by the visible light image acquisition device are transformed to the PT coordinates of the PT camera. This transformation relationship satisfies the following formula (9):

[0142]

[0143] Where matrix R is the inverse of the homography matrix H, matrix R is a 3x3 matrix, and matrix A represents [x, y, 1]. T .

[0144] In this way, by solving matrix R (i.e., solving the homography matrix H), the transformation relationship between the pixel coordinates in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera can be obtained. Specifically, matrix R can be solved by obtaining n (N*N = n, where N is an integer greater than 1) non-collinear points, corresponding to n feature point pairs. These n feature point pairs are formed by moving the PTZ camera N*N points to obtain the PT coordinates of n (N*N = n) PTZ cameras (e.g., obtaining the PT coordinates of 9 PTZ cameras at 9 points in a 3*3 grid), and the best matching feature points from the n visible light images acquired by the visible light image acquisition device at the same time point.

[0145] For example, the above n feature point pairs can be represented as shown in Table 1 below:

[0146] Table 1

[0147]

[0148] In some embodiments, a feature matching algorithm is used to obtain the optimal matching feature points of the visible light image acquired by the visible light image acquisition device. Specifically, feature matching is performed between the visible light image sequence acquired by the visible light image acquisition device and the visible light image sequence of the PTZ camera to obtain the optimal matching feature points of the visible light image acquired by the visible light image acquisition device.

[0149] Among them, the visible light image sequence of the PTZ camera includes multiple visible light images from the PTZ camera; the visible light image sequence acquired by the visible light image acquisition device includes multiple visible light images acquired by the visible light image acquisition device.

[0150] For example, the visible light image acquisition device of a PT camera performs PT movement to acquire nine visible light images from nine points in a 3x3 grid, forming a visible light image sequence of the PT camera; correspondingly, the visible light image acquisition device of a bullet camera can acquire nine visible light images acquired by the visible light image acquisition device at the same time point, forming a visible light image sequence acquired by the visible light image acquisition device.

[0151] For example, this feature matching algorithm can employ the Speeded-Up RobustFeatures (SURF) algorithm. SURF is a highly robust local feature point detector capable of feature point detection, feature point description, and feature point matching.

[0152] For example, taking SURF as an example, determining the feature points in the visible light image acquired by the visible light image acquisition device of the gun bolt can be specifically implemented as follows:

[0153] (1) Feature point detection

[0154] For each visible light image in the visible light image sequence of the PTZ camera and each visible light image acquired by the visible light image acquisition device in the visible light image sequence of the visible light image acquisition device, feature points are extracted from the visible light images of the PTZ camera and the visible light images acquired by the visible light image acquisition device, respectively, and the location information of the feature points is determined.

[0155] Specifically, the feature point detection steps are as follows: Gaussian smoothing is applied to both the visible light image from the PTZ camera and the visible light image acquired by the visible light image acquisition device, and a Gaussian scale space is established. Then, the location of the feature point is determined by calculating the local extrema of the determinant of the Hessian matrix. For any point in the space with scale α... The Hessian matrix is ​​defined by the following formula (10):

[0156]

[0157] Among them, L xx It is the second derivative of Gauss. The result of convolving with I = (x,y), where, L xy L yy They have the same meaning.

[0158] (2) Feature point description

[0159] Based on the location information of the feature points, the main direction of the feature points is calculated, the neighborhood information of the feature points is obtained, and the final feature point descriptor is obtained.

[0160] Specifically, after determining the location of the feature points, the principal direction of each feature point is calculated. The scale of the feature point is σ. Within a circular region with a radius of 6σ centered on the feature point, the image is processed using Haar wavelet response in the x and y directions. The Haar wavelet side length is 4σ, and a Gaussian weighting function with a scale of 2σ is used to Gaussian weight the Haar wavelet response values, so that the response contribution is greater closer to the feature point. Then, a sector of size π / 3 is used to traverse the entire circular region, and the direction of the maximum value of the vector sum and magnitude of the horizontal and vertical responses within the domain is defined as the principal direction of the feature point.

[0161] Then, a square window region with one side perpendicular to the main direction and a side length of 20σ is constructed with the feature point as the center. This window region is divided into 4×4 sub-regions. In each sub-region, the horizontal and vertical Haar wavelet responses of 5σ×5σ sampling points are calculated and denoted as dx and dy, respectively. Similarly, the Haar wavelet response values ​​are Gaussian weighted using a Gaussian weighting function with a scale of 2σ to increase the robustness to geometric transformation. Then, the response values ​​and absolute values ​​of each sub-region are added to form Σdx, Σdy, Σ|dx|, Σ|dy|. Thus, each sub-region forms a four-dimensional feature description vector V, which satisfies the following formula (11):

[0162] V=(∑dx,∑dy,∑|dx|,∑|dy) Formula (11)

[0163] For each feature point, a 4×4×4=64-dimensional feature vector is formed; finally, the feature vector is normalized to obtain the final SURF feature point descriptor, which contains neighborhood information of the feature point.

[0164] (3) Feature point matching

[0165] In some embodiments, a random sampling consensus algorithm is first used to remove unreliable feature points in the visible light image acquired by the visible light image acquisition device; then, feature points whose coordinates are less than a distance threshold α in the visible light image acquired by the visible light image acquisition device are taken as potential best matching feature points; furthermore, the feature point in the visible light image that is closest to the PTZ camera among the potential best matching feature points is taken as the best matching feature point in the visible light image acquired by the visible light image acquisition device.

[0166] For example, as shown in Table 2, based on the feature points of the visible light images from n PTZ cameras, the optimal matching feature points of the visible light images acquired by the n visible light image acquisition devices can be determined:

[0167] Table 2

[0168]

[0169] Therefore, by solving matrix R (i.e., solving the homography matrix H), the transformation relationship between the pixel coordinates in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera is determined, which is the second transformation relationship. For example, this second transformation relationship can satisfy the following formula (12):

[0170]

[0171] Where k is a constant, which can be determined by the zoom and tilt values ​​of the first calibration point; or k can use the default scaling factor; or k can be set by the user.

[0172] Specifically, by simultaneously solving the transformation relationships between the pixel coordinates and world coordinates in the visible light image acquired by the visible light image acquisition device, and between the world coordinates and the PTZ coordinates of the PTZ camera, the coordinate transformation relationship between the pixel coordinates and the PTZ coordinates in the visible light image acquired by the visible light image acquisition device can be obtained. For example, the second transformation relationship can also satisfy the following formula (13):

[0173]

[0174] S203. Based on the first transformation relationship and the second transformation relationship, determine the target transformation relationship.

[0175] The target transformation relationship refers to the coordinate transformation relationship between the pixel coordinates in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera.

[0176] In some embodiments, the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device (i.e., the above formula (1)) is substituted into the coordinate transformation relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera (i.e., the above formula (13)). This yields the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera (i.e., the coordinates of pixels in the visible light image acquired by the visible light image acquisition device in the second transformation relationship are replaced with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship to obtain the target transformation relationship). For example, the target transformation relationship can satisfy the following formula (14):

[0177]

[0178] Based on the technical solution provided in the embodiments of this application, at least the following beneficial effects can be achieved: According to the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device, and the coordinate transformation relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera, the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera is obtained, realizing automatic calibration between the bullet camera and the PTZ camera, eliminating the need for manual operation by the user, reducing manpower consumption, improving calibration efficiency, and avoiding the influence of subjective human factors, thus improving the accuracy of calibration.

[0179] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] This application embodiment can divide the gun-ball linkage device and calibration device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0181] like Figure 10 As shown in the figure, this application provides a camera-ball linkage device applied to a camera-ball linkage system. The camera-ball linkage system includes a first camera and a ball camera, and the first camera includes a thermal imaging image acquisition device. The camera-ball linkage device 300 includes an acquisition module 301, a conversion module 302, and a linkage module 303. In some embodiments, the camera-ball linkage device 300 further includes a calibration module 304.

[0182] The acquisition module 301 is used to acquire the pixel coordinates of the target object in the target thermal imaging image acquired by the thermal imaging image acquisition device.

[0183] The conversion module 302 is used to convert pixel coordinates into PTZ coordinates of the target object according to the target conversion relationship; the target conversion relationship is the coordinate conversion relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera.

[0184] The linkage module 303 is used to adjust the PTZ camera to a specified state based on the PTZ coordinates of the target object; wherein, the specified state is the state of the PTZ camera when the target object is located at a preset position within the field of view of the PTZ camera.

[0185] In some embodiments, the PTZ linkage system includes: at least one camera, the at least one camera including a thermal imaging image acquisition device and a visible light image acquisition device; a calibration module 304, configured to determine a first conversion relationship; wherein the first conversion relationship is a coordinate conversion relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device; determine a second conversion relationship; wherein the second conversion relationship is a coordinate conversion relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera; and determine a target conversion relationship based on the first conversion relationship and the second conversion relationship.

[0186] Optionally, the angle between the optical axis of the thermal imaging lens in the thermal imaging image acquisition device and the optical axis of the visible light lens in the visible light image acquisition device is less than a preset threshold.

[0187] In some embodiments, the first conversion relationship is characterized by a scaling factor and an offset factor between the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device; the scaling factor is the ratio of the width to the height of the thermal image acquired by the thermal imaging device and the visible light image acquired by the visible light image acquisition device; the offset factor is the coordinate difference between the coordinates of a pixel in the thermal image acquired by the thermal imaging device and the coordinates of the corresponding pixel in the visible light image acquired by the visible light image acquisition device; wherein, the pixels in the thermal image acquired by the thermal imaging device and the corresponding pixels in the visible light image acquired by the visible light image acquisition device represent pixels of the same object, or pixels representing the same part of the same object.

[0188] In some embodiments, the calibration module 304 is specifically used to determine a second conversion relationship based on the conversion relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the world coordinates, and the conversion relationship between the world coordinates and the PTZ coordinates of the PTZ camera.

[0189] In some embodiments, the calibration module 304 is specifically used to represent the coordinates of pixels in the visible light image acquired by the visible light image acquisition device with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and a first transformation relationship; and to replace the coordinates of pixels in the visible light image acquired by the visible light image acquisition device in the second transformation relationship with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship to obtain a target transformation relationship.

[0190] In some embodiments, the first gun bolt includes a visible light image acquisition device; or, at least one gun bolt further includes a second gun bolt, the second gun bolt including a visible light image acquisition device.

[0191] In some embodiments, the acquisition module 301 is further configured to receive the target transformation relationship.

[0192] like Figure 11 As shown in the figure, this application embodiment provides a calibration device applied to a PTZ camera system. The PTZ camera system includes at least one camera and one PTZ camera. The at least one camera includes a thermal imaging image acquisition device and a visible light image acquisition device. The calibration device 400 includes a calibration module 401.

[0193] The calibration module 401 is used to determine a first transformation relationship; wherein the first transformation relationship is the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the coordinates of pixels in the visible light image acquired by the visible light image acquisition device; determine a second transformation relationship; wherein the second transformation relationship is the coordinate transformation relationship between the coordinates of pixels in the visible light image acquired by the visible light image acquisition device and the PTZ coordinates of the PTZ camera; and based on the first and second transformation relationships, determine a target transformation relationship, wherein the target transformation relationship is the coordinate transformation relationship between the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the PTZ coordinates of the PTZ camera.

[0194] In some embodiments, the calibration module 401 is specifically used to represent the coordinates of pixels in the visible light image acquired by the visible light image acquisition device with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and a first transformation relationship; and to replace the coordinates of pixels in the visible light image acquired by the visible light image acquisition device in the second transformation relationship with the coordinates of pixels in the thermal imaging image acquired by the thermal imaging image acquisition device and the first transformation relationship to obtain a target transformation relationship.

[0195] In some embodiments, the at least one gun bolt includes: a first gun bolt, the first gun bolt including a thermal imaging image acquisition device and a visible light image acquisition device; or, the at least one gun bolt includes: a first gun bolt and a second gun bolt; wherein, the first gun bolt includes a thermal imaging image acquisition device; and the second gun bolt includes a visible light image acquisition device.

[0196] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a schematic diagram of the electronic device. Optionally, the electronic device can be a ball-and-shoot linkage device; or, the electronic device can be a calibration device. When the electronic device is a ball-and-shoot linkage device, it can perform the following functions: Figure 2 The gun-ball linkage method shown; when the electronic device is a calibration device, the electronic device can perform the following: Figure 4 The calibration method shown.

[0197] like Figure 12 As shown, the electronic device 500 includes: a processor 502, a communication interface 503, and a bus 504. Optionally, the electronic device 500 may also include a memory 501.

[0198] Processor 502 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 502 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 502 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0199] Communication interface 503 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0200] The memory 501 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0201] As one possible implementation, the memory 501 can exist independently of the processor 502. The memory 501 can be connected to the processor 502 via a bus 504 and is used to store instructions or program code. When the processor 502 calls and executes the instructions or program code stored in the memory 501, it can implement the gun-ball linkage method or calibration method provided in the embodiments of this application.

[0202] In another possible implementation, the memory 501 can also be integrated with the processor 502.

[0203] Bus 504 can be an extended industry standard architecture (EISA) bus, etc. Bus 504 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0204] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the gun-ball linkage device or calibration device can be divided into different functional modules to complete all or part of the functions described above.

[0205] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed on a computer, cause the computer to perform any of the methods provided in the above embodiments.

[0206] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to perform any of the methods provided in the above embodiments.

[0207] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0208] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0209] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A gunball linkage method, characterized by, The method is applied to a gun-ball linkage system, and the gun-ball linkage system comprises a first gun and a ball machine, the first gun comprises a thermal imaging image acquisition device, and the method comprises the following steps: Obtaining pixel coordinates of a target object in a target thermal imaging image collected by the thermal imaging image acquisition device in the target thermal imaging image; Converting the pixel coordinates into PTZ coordinates of the target object according to a target conversion relationship; the target conversion relationship is a coordinate conversion relationship between the coordinates of the pixels in the thermal imaging image collected by the thermal imaging image acquisition device and the PTZ coordinates of the ball machine; the target conversion relationship is determined based on a first conversion relationship and a second conversion relationship; the gun-ball linkage system comprises at least one gun, and the at least one gun comprises the thermal imaging image acquisition device and a visible light image acquisition device; wherein the first conversion relationship is a coordinate conversion relationship between the coordinates of the pixels in the thermal imaging image collected by the thermal imaging image acquisition device and the coordinates of the pixels in the visible light image collected by the visible light image acquisition device; the second conversion relationship is a coordinate conversion relationship between the coordinates of the pixels in the visible light image and the PTZ coordinates of the ball machine; Adjusting the ball machine to a specified state according to the PTZ coordinates of the target object; wherein the specified state is a state of the ball machine when the target object is located at a preset position in a field of view range of the ball machine.

2. The method of claim 1, wherein, The method further comprises: Determining the first conversion relationship; Determining the second conversion relationship; Determining the target conversion relationship based on the first conversion relationship and the second conversion relationship.

3. The method of claim 2, wherein, An included angle between an optical axis of a thermal imaging lens in the thermal imaging image acquisition device and an optical axis of a visible light lens in the visible light image acquisition device is less than a preset threshold value.

4. The method of claim 2, wherein, The first conversion relationship is characterized by a scaling coefficient and an offset coefficient between the thermal imaging image and the visible light image; The scaling coefficient is a ratio of widths and a ratio of heights of the thermal imaging image and the visible light image; the offset coefficient is a coordinate difference between the coordinates of the pixels in the thermal imaging image and the coordinates of corresponding pixels in the visible light image; wherein the pixels in the thermal imaging image and the corresponding pixels in the visible light image represent pixels of the same object or pixels of the same part of the same object; The determination of the second conversion relationship comprises: Determining the second conversion relationship according to a conversion relationship between the coordinates of the pixels in the visible light image and world coordinates and a conversion relationship between the world coordinates and the PTZ coordinates of the ball machine.

5. The method of claim 2, wherein, The determination of the target conversion relationship based on the first conversion relationship and the second conversion relationship comprises: Expressing the coordinates of the pixels in the visible light image by using the coordinates of the pixels in the thermal imaging image and the first conversion relationship; Replacing the coordinates of the pixels in the visible light image in the second conversion relationship with the coordinates of the pixels in the thermal imaging image and the first conversion relationship to obtain the target conversion relationship.

6. The method according to claim 2, wherein The first gun comprises the visible light image acquisition device. Or, the at least one gun also includes a second gun, and the second gun includes the visible light image acquisition device.

7. The method of claim 1, wherein, The method further includes: Receiving the target conversion relationship.

8. A calibration method characterized by, Applied to a gun-ball linkage system, the gun-ball linkage system includes at least one gun and a ball machine, the at least one gun includes a thermal imaging image acquisition device and a visible light image acquisition device, and the method includes: Determining a first conversion relationship, wherein the first conversion relationship is a coordinate conversion relationship between the coordinates of pixels in a thermal imaging image collected by the thermal imaging image acquisition device and the coordinates of pixels in a visible light image collected by the visible light image acquisition device; Determining a second conversion relationship, wherein the second conversion relationship is a coordinate conversion relationship between the coordinates of pixels in the visible light image and the PTZ coordinates of the ball machine; Based on the first conversion relationship and the second conversion relationship, determining a target conversion relationship, which is a coordinate conversion relationship between the coordinates of pixels in the thermal imaging image and the PTZ coordinates of the ball machine.

9. The method of claim 8, wherein, An included angle between an optical axis of a thermal imaging lens in the thermal imaging image acquisition device and an optical axis of a visible light lens in the visible light image acquisition device is less than a preset threshold value.

10. The method of claim 8, wherein, The determination of the target conversion relationship based on the first conversion relationship and the second conversion relationship includes: Expressing the coordinates of pixels in the visible light image with the coordinates of pixels in the thermal imaging image and the first conversion relationship; Replacing the coordinates of pixels in the visible light image in the second conversion relationship with the coordinates of pixels in the thermal imaging image and the first conversion relationship to obtain the target conversion relationship.

11. The method of claim 8, wherein: The at least one gun includes a first gun, and the first gun includes the thermal imaging image acquisition device and the visible light image acquisition device; Or, the at least one gun includes a first gun and a second gun, wherein the first gun includes the thermal imaging image acquisition device, and the second gun includes the visible light image acquisition device.

12. A gun ball linkage comprising: Applied to a gun-ball linkage system, the gun-ball linkage system includes a first gun and a ball machine, the first gun includes a thermal imaging image acquisition device, and the device includes: An acquisition module configured to acquire pixel coordinates of a target object in a target thermal imaging image collected by the thermal imaging image acquisition device in the target thermal imaging image; The conversion module is configured to convert the pixel coordinates into PTZ coordinates of the target object according to a target conversion relationship; the target conversion relationship is a coordinate conversion relationship between coordinates of pixels in a thermal imaging image collected by the thermal imaging image collection device and PTZ coordinates of the PTZ camera; the target conversion relationship is determined based on a first conversion relationship and a second conversion relationship; the gun-ball linkage system comprises at least one gun, and the at least one gun comprises the thermal imaging image collection device and a visible light image collection device; wherein the first conversion relationship is a coordinate conversion relationship between coordinates of pixels in a thermal imaging image collected by the thermal imaging image collection device and coordinates of pixels in a visible light image collected by the visible light image collection device; and the second conversion relationship is a coordinate conversion relationship between coordinates of pixels in the visible light image and the PTZ coordinates of the PTZ camera; The linkage module is configured to adjust the PTZ camera to a specified state according to the PTZ coordinates of the target object; wherein the specified state is a state of the PTZ camera when the target object is located at a preset position in a field of view of the PTZ camera.

13. The apparatus of claim 12, wherein: The apparatus further comprises a calibration module; the calibration module is configured to determine the first conversion relationship; determine the second conversion relationship; and determine the target conversion relationship based on the first conversion relationship and the second conversion relationship; The first conversion relationship is characterized by a scaling coefficient and an offset coefficient between the thermal imaging image and the visible light image; the scaling coefficient is a ratio of widths and a ratio of heights of the thermal imaging image and the visible light image; and the offset coefficient is a coordinate difference between coordinates of pixels in the thermal imaging image and coordinates of corresponding pixels in the visible light image; wherein the pixels in the thermal imaging image and the corresponding pixels in the visible light image represent pixels of the same object or pixels of the same part of the same object; The calibration module is specifically configured to determine the second conversion relationship according to a conversion relationship between coordinates of pixels in the visible light image and world coordinates and a conversion relationship between the world coordinates and the PTZ coordinates of the PTZ camera; The calibration module is specifically configured to represent coordinates of pixels in the visible light image by using coordinates of pixels in the thermal imaging image and the first conversion relationship; and replace coordinates of pixels in the visible light image in the second conversion relationship with coordinates of pixels in the thermal imaging image and the first conversion relationship to obtain the target conversion relationship; The first gun comprises the visible light image collection device; or the at least one gun further comprises a second gun, and the second gun comprises the visible light image collection device; The acquisition module is further configured to receive the target conversion relationship.

14. A calibration device, characterized by The apparatus is applied to a gun-ball linkage system, and the gun-ball linkage system comprises at least one gun and a PTZ camera; and the apparatus comprises: The calibration module is configured to determine a first conversion relationship, wherein the first conversion relationship is a coordinate conversion relationship between a coordinate of a pixel in a thermal image collected by the thermal image collection device and a coordinate of a pixel in a visible light image collected by the visible light image collection device; determine a second conversion relationship, wherein the second conversion relationship is a coordinate conversion relationship between the coordinate of the pixel in the visible light image and a PTZ coordinate of the PTZ camera; and determine a target conversion relationship based on the first conversion relationship and the second conversion relationship, wherein the target conversion relationship is a coordinate conversion relationship between the coordinate of the pixel in the thermal image and the PTZ coordinate of the PTZ camera.

15. The apparatus of claim 14, wherein, The calibration module is specifically configured to express the coordinate of the pixel in the visible light image by using the coordinate of the pixel in the thermal image and the first conversion relationship; and replace the coordinate of the pixel in the visible light image in the second conversion relationship by the coordinate of the pixel in the thermal image and the first conversion relationship to obtain the target conversion relationship. The at least one gun includes a first gun, and the first gun includes the thermal image collection device and the visible light image collection device; or the at least one gun includes a first gun and a second gun, wherein the first gun includes the thermal image collection device, and the second gun includes the visible light image collection device.

16. A gun ball linkage comprising: The apparatus includes: one or more processors; one or more memories; The one or more memories are configured to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the gun-ball linkage device executes the gun-ball linkage method of any one of claims 1 to 7.

17. A calibration device, characterized by The apparatus includes: one or more processors; one or more memories; The one or more memories are configured to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the calibration device executes the calibration method of any one of claims 8 to 11.

Images