Panoramic stitching method and system based on holder equipment

Abstract

The invention discloses a panoramic stitching method and system based on pan-tilt equipment. The panoramic stitching method comprises the following steps: determining a starting point and an ending point of panoramic stitching in a rectangular bounding box of an image containing a target; acquiring a screenshot moving track; the holder is controlled to move and move along the capture movement track until a movement end point position is reached, and when the rectangular view field frame is located at the movement starting position, each time the rectangular view field frame moves and is located at the movement end point position, a corresponding image is obtained through an imaging device connected with the holder; and splicing the images acquired by all the imaging devices to acquire a panoramic spliced image. According to the invention, the pan-tilt device drives the imaging device, performs timed image capturing according to the automatically planned image capturing path, performs splicing, finally obtains the complete features of a large target, prevents image feature points from being lost or cut due to dislocation, can select dual-light panoramic splicing or infrared image splicing according to the requirements of a splicing scene, and improves the splicing efficiency. Therefore, the application range is expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a panorama stitching method and system based on a pan / tilt device. Background Art

[0002] Panoramic stitching is used to stitch multiple images into a large-scale image. Its main process includes: acquiring multiple images of the same plane in space through the same camera, and then using the Gaussian difference (DoG) key point detector and SIFT feature descriptor to obtain key points and features, and performing feature matching through the K-nearest neighbor algorithm. Then, based on at least four matching key points, an algorithm is used to estimate the projection transformation matrix, and finally the images are stitched together after projection transformation to generate a panoramic image.

[0003] Since panoramic stitching technology is often completed through mobile devices such as mobile phones and handheld devices, and factors such as the displacement method and displacement distance of these devices will change each time, resulting in adjacent pictures not being in the same horizontal or vertical direction, the feature association key points between pictures are lost, which in turn causes the pictures to be cropped or deformed in large quantities, ultimately affecting the image stitching effect. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a panoramic stitching method and system based on a pan-tilt device, which drives an imaging device through the pan-tilt device, captures images at regular intervals according to an automatically planned capture path, and performs stitching, so as to finally obtain the complete features of a larger target, thereby preventing image feature points from being lost or cropped due to misalignment, and can select dual-light panoramic stitching or infrared image stitching according to the needs of the stitching scene, thereby expanding the scope of application.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] On the one hand, a panorama stitching method based on a pan / tilt device is provided, which comprises the following steps:

[0007] Acquire an image containing the target, and determine a starting point and an ending point of the panoramic stitching within a rectangular bounding box of the image containing the target;

[0008] Acquire the capture movement trajectory according to the starting point and the end point of the panoramic stitching and the total number of times the rectangular field of view frame of the imaging device moves in the horizontal and vertical directions respectively;

[0009] Controlling the movement of the pan-tilt head so that the rectangular field of view frame of the imaging device starts from the moving starting position and moves along the image capturing moving trajectory until it reaches the moving ending position, and when the rectangular field of view frame is at the moving starting position, each time it moves, and at the moving ending position, a corresponding image is acquired through the imaging device connected to the pan-tilt head;

[0010] The images acquired by all imaging devices are stitched together to obtain a panoramic stitched image.

[0011] Preferably, determining the starting point and the ending point of the panoramic stitching within the rectangular bounding box comprises the following steps:

[0012] Generate a rectangular field of view frame in an imaging device, and drive the imaging device to move by a pan / tilt platform so that the rectangular field of view frame moves synchronously within a rectangular boundary frame;

[0013] When the imaging device moves to a point where the first vertex of the rectangular field of view frame coincides with the first vertex of the rectangular bounding box, and the center of the rectangular field of view frame is located within the rectangular bounding box, the current gimbal coordinates are used as the starting point of the panoramic stitching;

[0014] When the imaging device moves to the point where the second vertex of the rectangular field of view coincides with the second vertex of the rectangular bounding box, and the center of the rectangular field of view is located within the rectangular bounding box, the current gimbal coordinates are used as the end point of the panoramic stitching.

[0015] Preferably, the first vertex and the second vertex of the rectangular bounding box are respectively two endpoints of the same diagonal line of the rectangular bounding box; the first vertex and the second vertex of the rectangular field of view box are respectively two endpoints of the same diagonal line of the rectangular field of view box; and the diagonal line connecting the first vertex and the second vertex of the rectangular field of view box is collinear with the diagonal line connecting the first vertex and the second vertex of the rectangular bounding box.

[0016] Preferably, obtaining the capture movement trajectory according to the starting point and the end point of the panoramic stitching and the total number of times the rectangular field of view frame of the imaging device moves in the horizontal and vertical directions respectively comprises the following steps:

[0017] Divide the rectangular bounding box into a plurality of squares, and make the center of the first diagonal square coincide with the starting point of the panoramic stitching, the center of the second diagonal square coincide with the ending point of the panoramic stitching, and the centers of the first diagonal square and the second diagonal square are located on the same diagonal line;

[0018] The imaging device is driven to move by the pan / tilt platform so that the rectangular field of view of the imaging device completely / partially overlaps with the first diagonal square;

[0019] Obtain the total number of horizontal and vertical movements of the rectangular field of view frame in the horizontal and vertical directions respectively;

[0020] A horizontal moving path of the rectangular field of view frame is constructed according to the total number of horizontal movements and the horizontal unit moving distance, and a vertical moving path of the rectangular field of view frame is constructed according to the total number of vertical movements and the vertical unit moving distance, and the horizontal moving path and the vertical moving path are connected in sequence to obtain the capture moving trajectory.

[0021] Preferably, the starting point and the end point of the capture movement trajectory are respectively located in the first diagonal grid and the second diagonal grid, and the capture movement trajectory covers the starting point and the end point of the panoramic stitching.

[0022] Preferably, the total number of horizontal movements and the total number of vertical movements of the rectangular field of view frame in the horizontal direction and the vertical direction are obtained by the following formulas respectively:

[0023]

[0024] Among them, m and n are the total number of horizontal movements and the total number of vertical movements respectively; a is the horizontal unit movement distance of the rectangular field of view frame; b is the vertical unit movement distance of the rectangular field of view frame; Represents the smallest integer not less than x; x0 and y0 are the horizontal and vertical coordinates of the starting point of panoramic stitching respectively; x1 and y1 are the horizontal and vertical coordinates of the ending point of panoramic stitching respectively.

[0025] Preferably, the horizontal moving path is divided into a plurality of horizontal moving path units, and the units are distributed in different rows, and there is only one horizontal moving path unit in each row, and the moving directions of the horizontal moving path units in two adjacent rows are opposite;

[0026] The vertical movement path is divided into a plurality of vertical movement path units, and the units are distributed in different columns, and there is only one vertical movement path unit in the column, and the movement directions of all the vertical movement path units are the same;

[0027] Two adjacent horizontal movement path units are connected via a vertical movement path unit, and all horizontal movement path units and all vertical movement path units are connected end to end in sequence to obtain the capture movement trajectory.

[0028] Preferably, controlling the movement of the pan-tilt head so that the rectangular field of view frame of the imaging device starts from the moving starting position and moves along the capture moving trajectory until it reaches the moving end position, and when the rectangular field of view frame is located at the moving starting position, each time it moves, and at the moving end position, the corresponding image is acquired through the imaging device connected to the pan-tilt head, including the following steps:

[0029] Control the movement of the gimbal so that the midpoint of the rectangular field of view is located in the first diagonal square and on the capture movement trajectory, and obtain the image of the corresponding square through the imaging device connected to the gimbal;

[0030] Controlling the movement of the gimbal so that the midpoint of the rectangular field of view moves along a horizontal movement path passing through the first row of squares, and each time the horizontal movement occurs, the center of the rectangular field of view moves synchronously to a different square in the first row, and an imaging device that moves synchronously with the gimbal acquires an image in the corresponding square;

[0031] Controlling the movement of the pan-tilt head so that the midpoint of the rectangular field of view moves vertically along a vertical movement path to a square in the same column and in the next row, and acquiring an image in the square through an imaging device that moves synchronously with the pan-tilt head;

[0032] Repeat the above movement process so that the midpoint of the rectangular field of view moves along the capture movement trajectory, and after each movement, the image in the corresponding square is acquired by the imaging device that moves synchronously with the pan / tilt head;

[0033] When the midpoint of the rectangular field of view frame is located in the second diagonal square and on the capture movement trajectory, the image of the corresponding square is acquired through the imaging device connected to the pan / tilt head.

[0034] Preferably, the width and height of the panoramic stitching image meet the following conditions:

[0035]

[0036] Among them, α and β are the horizontal and vertical field of view angles of the imaging device respectively; W 原 , H 原 W are the width and height of the image obtained by the imaging device respectively; 全景 , H 全景 are the width and height of the panoramic stitching image respectively.

[0037] On the other hand, a panoramic stitching system is also provided, comprising:

[0038] Imaging equipment;

[0039] A stitching starting point determination unit, which is used to generate a rectangular bounding box surrounding the target in the image containing the target, and determine the starting point and the ending point of the panoramic stitching in the rectangular bounding box;

[0040] A pan / tilt platform connected to an imaging device to drive the imaging device to move;

[0041] A capture movement trajectory acquisition unit, which is used to acquire the capture movement trajectory according to the starting point and the ending point of the panoramic stitching;

[0042] A pan / tilt control unit connected to the pan / tilt and used to control the movement of the pan / tilt; the pan / tilt drives the imaging device to move, and when the rectangular field of view frame of the imaging device is at the starting position of the movement, after each movement along the capture movement trajectory, and at the end position of the movement, the image of the corresponding square is acquired through the imaging device connected to the pan / tilt;

[0043] And an image stitching unit, which is connected to the imaging device and is used to stitch all the images acquired by the imaging device connected to the pan-tilt head according to an image panoramic stitching algorithm to obtain a panoramic stitching image.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] In the present invention, the imaging device is driven by the gimbal device, and images are captured one by one through the rectangular field of view frame, and the images are stitched, cropped and optimized, so that the complete features of the larger target can be observed in one picture. In the process of capturing images, the stable and precise movement of the gimbal can ensure that the feature points of the image will not be lost or cropped due to misalignment. The images can be captured at regular intervals according to the automatically planned capturing path and stitched, thereby releasing manpower, and can be used as an API for regular calls. At the same time, dual-light panoramic stitching or infrared image stitching can be selected according to the needs of the stitching scene to expand the scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the steps of the panoramic stitching method based on the pan-tilt device in the present invention;

[0047] Figure 2 It is a position relationship diagram of the rectangular boundary box C0 and the rectangular field of view box C1 in the present invention;

[0048] Figure 3 It is a schematic diagram of the trajectory of the rectangular field of view frame C1 when it moves along the capture movement trajectory in the present invention;

[0049] Figure 4 is the infrared panoramic stitching image obtained in the present invention;

[0050] Figure 5 It is a structural schematic diagram of the panoramic stitching system based on the pan-tilt device in the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Embodiment 1:

[0053] like Figure 1 As shown, this embodiment provides a panorama stitching method based on a pan-tilt device, which includes the following steps:

[0054] S1, acquiring an image containing a target, and determining a starting point P0 (x0, y0) of panoramic stitching and an ending point P1 (x1, y1) of panoramic stitching within a rectangular bounding box C0 of the image containing the target; wherein the image containing the target may be a visible light image;

[0055] In this embodiment, the starting point P0 (x0, y0) of the panoramic stitching and the ending point P1 (x1, y1) of the panoramic stitching are located on the same diagonal line of the rectangular bounding box C0, and the starting point P0 (x0, y0) of the panoramic stitching and the ending point P1 (x1, y1) of the panoramic stitching are determined within the rectangular bounding box C0, including the following steps:

[0056] Generate a rectangular field of view frame C1 in an imaging device (such as an infrared camera), and drive the imaging device to move through a pan / tilt platform so that the rectangular field of view frame C1 moves synchronously within a rectangular boundary frame C0;

[0057] When the imaging device moves to the point where the first vertex C11 of the rectangular field of view frame C1 coincides with the first vertex C01 of the rectangular bounding frame C0, and the center of the rectangular field of view frame C1 is located within the rectangular bounding frame C0, the current gimbal coordinates are used as the starting point P0 (x0, y0) of the panoramic stitching;

[0058] When the imaging device moves to the point where the second vertex C12 of the rectangular field of view frame C1 coincides with the second vertex C02 of the rectangular bounding frame C0, and the center of the rectangular field of view frame C1 is located within the rectangular bounding frame C0, the current gimbal coordinates are used as the end point P1 (x1, y1) of the panoramic stitching;

[0059] The first vertex C01 and the second vertex C02 of the rectangular bounding box C0 are respectively two end points of the same diagonal line of the rectangular bounding box C0; the first vertex C11 and the second vertex C12 of the rectangular field of view box C1 are respectively two end points of the same diagonal line of the rectangular field of view box C1; and the diagonal line connecting the first vertex C11 and the second vertex C12 of the rectangular field of view box C1 is collinear with the diagonal line connecting the first vertex C01 and the second vertex C02 of the rectangular bounding box C0;

[0060] S2, obtaining the capture movement trajectory according to the starting point P0 (x0, y0) and the ending point P1 (x1, y1) of the panoramic stitching, which specifically includes the following steps:

[0061] like Figure 3As shown, the rectangular bounding box C0 is divided into p*q (p=m+1, q=n+1) squares, where p and q are rows and columns, respectively, and are positive integers greater than or equal to 2, and p and q may be the same or different, for example, in this embodiment, p=q=3; and the center of the first diagonal square s1 coincides with the starting point P0 (x0, y0) of the panoramic stitching, the center of the second diagonal square s2 coincides with the ending point P1 (x1, y1) of the panoramic stitching, and the centers of the first diagonal square s1 and the second diagonal square s2 are located on the same diagonal line. For example, in this embodiment, the centers of the upper left corner square (i.e., the first diagonal square s1) and the lower right corner square (i.e., the second diagonal square s2) coincide with the starting points P0 (x0, y0) and P1 (x1, y1) of the panoramic stitching, respectively, and the centers of the upper left corner square and the lower right corner square are both located on the same diagonal line;

[0062] The imaging device is driven to move by the pan / tilt platform so that the rectangular field of view frame C1 of the imaging device completely / partially overlaps with the first diagonal square s1. Preferably, the centers of the rectangular field of view frame C1 and the first diagonal square s1 overlap;

[0063] The total number of horizontal movements m and the total number of vertical movements n of the rectangular field of view frame C1 in the horizontal direction (i.e., the column direction) and the vertical direction (i.e., the row direction) are obtained respectively, where:

[0064]

[0065] Wherein, a is the horizontal unit moving distance of the rectangular field of view frame C1; b is the vertical unit moving distance of the rectangular field of view frame C1; represents the smallest integer not less than x, for example, x = 3, then If x = 3.5, then

[0066] The movement refers to driving the pan / tilt and the imaging device connected to the pan / tilt to move synchronously by rotating the pan / tilt motion drive unit (such as a drive motor), so that the rectangular field of view frame C1 moves in the horizontal direction and the vertical direction. The horizontal unit movement distance and the vertical unit movement distance are the unit movement distances of the rectangular field of view frame C1 in the horizontal direction and the vertical direction when the pan / tilt motion drive unit (such as a drive motor) is rotated each time. At the same time, after the ratio of the distance change amount to the unit movement distance is rounded up, the total number of horizontal movements m and the total number of vertical movements n finally obtained can ensure that the field of view of the imaging device can completely include the entire target;

[0067] A horizontal moving path of the rectangular field of view frame C1 is constructed according to the total number of horizontal movements m and the horizontal unit moving distance, and a vertical moving path of the rectangular field of view frame C1 is constructed according to the total number of vertical movements n and the vertical unit moving distance, and the horizontal moving path and the vertical moving path are sequentially connected to obtain a non-closed, unidirectional capture moving trajectory, and the capture moving trajectory passes through each square sequentially;

[0068] The starting point and the end point of the capture movement trajectory are respectively located in the first diagonal square s1 and the second diagonal square s2, and the capture movement trajectory covers the starting point P0 (x0, y0) and the end point P1 (x1, y1) of the panoramic stitching;

[0069] In this embodiment, the horizontal movement path can be divided into a plurality of horizontal movement path units H1, and the horizontal movement path units H1 are distributed in different rows, and there is only one horizontal movement path unit H1 in each row. The movement directions of the horizontal movement path units H1 in two adjacent rows are opposite. The length of each horizontal movement path unit H1 = the number of horizontal movements of the pan / tilt in the row × the horizontal unit movement distance;

[0070] Similarly, the vertical movement path can be divided into a number of vertical movement path units V1, and distributed in different columns, and there is only one vertical movement path unit V1 in the column, and the movement direction of all vertical movement path units V1 is the same, and the length of each vertical movement path unit V1 = vertical unit movement distance;

[0071] Two adjacent horizontal moving path units H1 are connected by a vertical moving path unit V1, and all horizontal moving path units H1 and all vertical moving path units V1 are connected end to end in sequence to obtain a non-closed capture moving trajectory;

[0072] For example, Figure 3 As shown, when p=q=3, the rectangular boundary box C0 is divided into 3*3 squares, which are numbered 1, 2, 3, ..., 9 in sequence, and the horizontal movement path can be divided into 3 horizontal movement path units H1, which are respectively distributed in the 1st to 3rd rows. If the number of horizontal movements of the rectangular field of view box C1 in each row is 2 times (m=6), the length of each horizontal movement path unit H1=2×horizontal unit movement distance, and the horizontal movement path units H1 in the 1st to 2nd rows and the 2nd to 3rd rows move in opposite directions;

[0073] The vertical movement path can be divided into two vertical movement path units V1, which are correspondingly distributed in the first column and the third column (n=2), and the movement directions of the two vertical movement path units V1 are the same, and the length of each vertical movement path unit V1 is equal to the vertical unit movement distance;

[0074] Then, the vertical moving path unit V1 in the first column is connected to the horizontal moving path unit H1 in the first and second rows, and the vertical moving path unit V1 in the third column is connected to the horizontal moving path unit H1 in the second and third rows, so as to obtain the capture moving track;

[0075] S3, controlling the movement of the pan / tilt platform, thereby driving the imaging device to move, so that the rectangular field of view frame C1 starts from the moving starting position and moves along the capture moving trajectory until it reaches the moving end position, and each time the rectangular field of view frame C1 moves, its center synchronously moves to a different square;

[0076] When the rectangular field of view frame C1 is located at the moving starting position, each time it moves, and at the moving end position, an image of the corresponding square is acquired through an imaging device connected to the pan / tilt head, the image is a Y16 infrared image and / or a visible light image, and the overlap rate of similar features of two adjacent images acquired is greater than or equal to 30%, and the overlap rate of similar features can be determined by the horizontal unit moving distance a and the vertical unit moving distance b of the rectangular field of view frame C1;

[0077] Specifically, step S3 includes the following steps:

[0078] Control the movement of the pan / tilt, thereby driving the imaging device to move, so that the midpoint of the rectangular field of view frame C1 is located in the first diagonal square s1 and on the capture movement trajectory. At this time, it is considered that the rectangular field of view frame C1 is located at the starting position of the movement, and the image of the corresponding square (that is, the first diagonal square s1) is acquired through the imaging device connected to the pan / tilt. Preferably, the midpoint of the rectangular field of view frame C1 is located in the first diagonal square s1 and coincides with the starting point of the capture movement trajectory;

[0079] The pan / tilt movement is controlled, thereby driving the imaging device to move, so that the midpoint of the rectangular field of view frame C1 moves horizontally m times along the horizontal movement path passing through the first row of squares, and each time the horizontal movement is performed, the center of the rectangular field of view frame C1 moves synchronously to different squares in the first row, and the imaging device that moves synchronously with the pan / tilt obtains the image in the corresponding square; and each time the horizontal movement is performed, the moving distance = the horizontal unit moving distance × the horizontal moving step length. In this embodiment, the horizontal moving step length is the horizontal moving distance of the rectangular field of view frame C1 when the pan / tilt movement driving unit rotates the first preset angle α (i.e. Figure 3 "horizontal α°" in the figure), wherein the value range of the first preset angle α is 5-15°;

[0080] The pan / tilt is controlled to move, thereby driving the imaging device to move, so that the midpoint of the rectangular field of view frame C1 moves vertically along a vertical movement path to a square in the same column and in the next row, and the image in the square is acquired by the imaging device that moves synchronously with the pan / tilt; and when moving vertically, the moving distance = vertical unit moving distance × vertical moving step length. In this embodiment, the vertical moving step length is the vertical moving distance of the rectangular field of view frame C1 when the pan / tilt motion driving unit rotates the second preset angle β (i.e. Figure 3 "vertical-β°" in the figure), wherein the value range of the second preset angle β is 5-15°;

[0081] Repeat the above-mentioned process of moving the midpoint of the rectangular field of view frame C1 along the horizontal moving path in the same row of squares, and the process of moving along the vertical moving path in the same column of squares, so that the midpoint of the rectangular field of view frame C1 moves along the capture moving trajectory, and after each movement, the image in the corresponding square is acquired by the imaging device that moves synchronously with the pan / tilt head;

[0082] Until the midpoint of the rectangular field of view frame C1 is located in the second diagonal square s2 and on the capture movement trajectory, it is considered that the rectangular field of view frame C1 is located at the movement end position, and the image of the corresponding square (that is, the second diagonal square s2) is acquired by an imaging device connected to the pan / tilt head. Preferably, the midpoint of the rectangular field of view frame C1 is located in the second diagonal square s2 and coincides with the end point of the capture movement trajectory;

[0083] Therefore, when the rectangular field of view frame C1 moves, its center is always on the capture movement track, and it moves horizontally and vertically along the capture movement track, and a total of p*q images are obtained in sequence;

[0084] For example, Figure 3 As shown, first, the rectangular field of view frame C1 is located at position 1 (i.e., the starting position of the movement), so that its center coincides with the starting point of the capture movement trajectory, and then the center of the rectangular field of view frame C1 moves horizontally twice along the horizontal movement path unit H1 of the first row, passes through position 2 and arrives at position 3, and then moves downward once along the vertical movement path unit V1 of the third column to arrive at position 4, takes position 4 as the starting point, moves horizontally twice along the horizontal movement path unit H1 of the second row, passes through position 5 and arrives at position 6, and then moves downward once along the vertical movement path unit V1 of the first column to arrive at position 7, takes position 7 as the starting point, moves horizontally twice along the horizontal movement path unit H1 of the third row, passes through position 8 and arrives at position 9 (i.e., the end position of the movement), and all the movements are completed at this time; in the above process, when the rectangular field of view frame C1 is located at the starting position of the movement, each time it moves (including horizontal movement and vertical movement) and at the end position of the movement, the imaging device correspondingly acquires an image, thereby obtaining a total of 3*3=9 images;

[0085] S4, sending all images acquired by the imaging device connected to the PTZ to the image stitching unit, calling the image panoramic stitching algorithm for stitching, so as to obtain a panoramic stitching image;

[0086] Specifically, the width and height of the panoramic stitching image meet the following conditions:

[0087]

[0088] Among them, α and β are the horizontal and vertical field of view angles of the imaging device respectively; W 原 , H 原 W are the width and height of the image obtained by the imaging device respectively; 全景 , H 全景 are the width and height of the panoramic stitching image respectively, which can ensure that the final panoramic stitching image can fully display the whole scene.

[0089] Since the CMOS of the infrared camera is relatively small, the observation range and effect of a single picture are greatly limited. In actual use, it is often the case that one infrared image cannot fully observe the complete infrared features of the object being measured. Therefore, in this embodiment, the target is first framed by a rectangular bounding box, and then the imaging device is driven by the pan-tilt device to capture pictures one by one through the rectangular field of view frame, and the images are spliced, cropped and optimized, and finally the complete features of the larger target (such as the infrared information of the target, etc.) are observed in one picture.

[0090] Furthermore, during the capture process, the movement of the pan / tilt is more stable and accurate than the movement of the imaging device under manual control, so that when capturing images, the feature points of the image will not be lost or cropped due to misalignment, and the images can be captured regularly according to the automatically planned capture path (i.e., the capture movement trajectory) and stitched, thereby releasing manpower and being used as an API for regular calls. At the same time, the captured image in this embodiment (i.e., the image obtained in step S3) can be a visible light image and an infrared image, further realizing dual-light panoramic stitching, or only the Y16 infrared image can be captured, and the infrared image obtained by stitching still has the temperature matrix feature (such as Figure 4 As shown, T1 is the grid image obtained by the imaging device when the rectangular field of view frame C1 moves, and T2 is the infrared image finally obtained by panoramic stitching). Infrared imaging and temperature measurement can be performed to expand the scope of application.

[0091] Embodiment 2:

[0092] This embodiment provides a panoramic stitching system, which can implement the panoramic stitching method described in Embodiment 1. Figure 5 As shown, it includes:

[0093] Imaging device 1;

[0094] A stitching starting point determination unit 2 is used to generate a rectangular bounding box C0 surrounding the target in the image containing the target, and determine the starting point P0 (x0, y0) of the panoramic stitching and the ending point P1 (x1, y1) of the panoramic stitching in the rectangular bounding box C0, and the specific process is the same as step S1;

[0095] A pan / tilt platform 3 connected to the imaging device 1 to drive the imaging device 1 to move;

[0096] The capture movement trajectory acquisition unit 4 is used to acquire the capture movement trajectory according to the starting point P0 (x0, y0) and the ending point P1 (x1, y1) of the panoramic stitching, and the specific process is the same as step S2;

[0097] A pan / tilt control unit 5, which is connected to the pan / tilt 3 and is used to control the movement of the pan / tilt 3; the pan / tilt 3 drives the imaging device 1 to move, and when the rectangular field of view frame C1 of the imaging device 1 is located at the starting position of the movement, after each movement along the capture movement trajectory, and at the end position of the movement, the image of the corresponding square is acquired through the imaging device connected to the pan / tilt;

[0098] And an image stitching unit 6, which is connected to the imaging device 1 and is used to stitch all the images sequentially acquired by the imaging devices connected to the pan-tilt head according to an image panoramic stitching algorithm to obtain a panoramic stitching image.

[0099] In summary, the present invention drives the imaging device through the gimbal device, captures images one by one through the rectangular field of view frame, and stitches, crops and optimizes the images, and finally observes the complete features of a larger target (such as the target's infrared information, etc.) in one picture.

[0100] Furthermore, during the capture process, the stable and precise movement of the gimbal can ensure that the feature points of the image will not be lost or cropped due to misalignment. The capture can be scheduled according to the automatically planned capture path and stitched to free up manpower. It can also be used as an API for scheduled calls. At the same time, dual-light panoramic stitching or infrared image stitching can be selected according to the needs of the stitching scene to expand the scope of application.

[0101] It should be noted that, in this article, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0102] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A panorama stitching method based on a pan-tilt device, characterized in that: The steps include: Acquire an image containing the target, and determine a starting point and an ending point of the panoramic stitching within a rectangular bounding box of the image containing the target; Acquire the capture movement trajectory according to the starting point and the end point of the panoramic stitching and the total number of times the rectangular field of view frame of the imaging device moves in the horizontal and vertical directions respectively; Controlling the movement of the pan-tilt head so that the rectangular field of view frame of the imaging device starts from the moving starting position and moves along the image capturing moving trajectory until it reaches the moving ending position, and when the rectangular field of view frame is at the moving starting position, each time it moves, and at the moving ending position, a corresponding image is acquired through the imaging device connected to the pan-tilt head; The images acquired by all imaging devices are stitched together to obtain a panoramic stitched image.

2. The panoramic stitching method according to claim 1, characterized in that: Determining the starting point and the ending point of the panoramic stitching within the rectangular bounding box includes the following steps: Generate a rectangular field of view frame in an imaging device, and drive the imaging device to move by a pan / tilt platform so that the rectangular field of view frame moves synchronously within a rectangular boundary frame; When the imaging device moves to a point where the first vertex of the rectangular field of view frame coincides with the first vertex of the rectangular bounding box, and the center of the rectangular field of view frame is located within the rectangular bounding box, the current gimbal coordinates are used as the starting point of the panoramic stitching; When the imaging device moves to the point where the second vertex of the rectangular field of view coincides with the second vertex of the rectangular bounding box, and the center of the rectangular field of view is located within the rectangular bounding box, the current gimbal coordinates are used as the end point of the panoramic stitching.

3. The panoramic stitching method according to claim 2, characterized in that: The first vertex and the second vertex of the rectangular bounding box are respectively two endpoints of the same diagonal line of the rectangular bounding box; the first vertex and the second vertex of the rectangular field of view box are respectively two endpoints of the same diagonal line of the rectangular field of view box; and the diagonal line connecting the first vertex and the second vertex of the rectangular field of view box is collinear with the diagonal line connecting the first vertex and the second vertex of the rectangular bounding box.

4. The panoramic stitching method according to claim 1, characterized in that: The capture movement trajectory is obtained according to the starting point and the end point of the panoramic stitching and the total number of times the rectangular field of view frame of the imaging device moves in the horizontal and vertical directions, including the following steps: Divide the rectangular bounding box into a plurality of squares, and make the center of the first diagonal square coincide with the starting point of the panoramic stitching, the center of the second diagonal square coincide with the ending point of the panoramic stitching, and the centers of the first diagonal square and the second diagonal square are located on the same diagonal line; The imaging device is driven to move by the pan / tilt platform so that the rectangular field of view of the imaging device completely / partially overlaps with the first diagonal square; Obtain the total number of horizontal and vertical movements of the rectangular field of view frame in the horizontal and vertical directions respectively; A horizontal moving path of the rectangular field of view frame is constructed according to the total number of horizontal movements and the horizontal unit moving distance, and a vertical moving path of the rectangular field of view frame is constructed according to the total number of vertical movements and the vertical unit moving distance, and the horizontal moving path and the vertical moving path are connected in sequence to obtain the capture moving trajectory.

5. The panoramic stitching method according to claim 4, characterized in that: The starting point and the end point of the capture movement trajectory are respectively located in the first diagonal grid and the second diagonal grid, and the capture movement trajectory covers the starting point and the end point of the panoramic stitching.

6. The panoramic stitching method according to claim 4, characterized in that: The total number of horizontal and vertical movements of the rectangular field of view frame in the horizontal and vertical directions are obtained by the following formulas respectively: Among them, m and n are the total number of horizontal movements and the total number of vertical movements respectively; a is the horizontal unit movement distance of the rectangular field of view frame; b is the vertical unit movement distance of the rectangular field of view frame; Represents the smallest integer not less than x; x0 and y0 are the horizontal and vertical coordinates of the starting point of panoramic stitching respectively; x1 and y1 are the horizontal and vertical coordinates of the ending point of panoramic stitching respectively.

7. The panoramic stitching method according to claim 4, characterized in that: The horizontal moving path is divided into a plurality of horizontal moving path units, which are distributed in different rows, and there is only one horizontal moving path unit in each row, and the moving directions of the horizontal moving path units in two adjacent rows are opposite; The vertical movement path is divided into a plurality of vertical movement path units, and the units are distributed in different columns, and there is only one vertical movement path unit in the column, and the movement directions of all the vertical movement path units are the same; Two adjacent horizontal movement path units are connected via a vertical movement path unit, and all horizontal movement path units and all vertical movement path units are connected end to end in sequence to obtain the capture movement trajectory.

8. The panoramic stitching method according to claim 4, characterized in that: Controlling the movement of the pan-tilt head so that the rectangular field of view frame of the imaging device starts from the moving starting position and moves along the capture moving trajectory until it reaches the moving end position, and when the rectangular field of view frame is at the moving starting position, each time it moves, and at the moving end position, the corresponding image is acquired through the imaging device connected to the pan-tilt head, including the following steps: Control the movement of the gimbal so that the midpoint of the rectangular field of view is located in the first diagonal square and on the capture movement trajectory, and obtain the image of the corresponding square through the imaging device connected to the gimbal; Controlling the movement of the gimbal so that the midpoint of the rectangular field of view moves along a horizontal movement path passing through the first row of squares, and each time the horizontal movement occurs, the center of the rectangular field of view moves synchronously to a different square in the first row, and an imaging device that moves synchronously with the gimbal acquires an image in the corresponding square; Controlling the movement of the pan-tilt head so that the midpoint of the rectangular field of view moves vertically along a vertical movement path to a square in the same column and in the next row, and acquiring an image in the square through an imaging device that moves synchronously with the pan-tilt head; Repeat the above movement process so that the midpoint of the rectangular field of view moves along the capture movement trajectory, and after each movement, the image in the corresponding square is acquired by the imaging device that moves synchronously with the pan / tilt head; When the midpoint of the rectangular field of view frame is located in the second diagonal square and on the capture movement trajectory, the image of the corresponding square is acquired through the imaging device connected to the pan / tilt head.

9. The panoramic stitching method according to claim 6, characterized in that: The width and height of the panoramic stitching image meet the following conditions: Among them, α and β are the horizontal and vertical field of view angles of the imaging device respectively; W 原 , H 原 W are the width and height of the image obtained by the imaging device respectively; 全景 , H 全景 are the width and height of the panoramic stitching image respectively.

10. A panoramic stitching system, characterized in that: include: Imaging equipment; A stitching starting point determination unit, which is used to generate a rectangular bounding box surrounding the target in the image containing the target, and determine the starting point and the ending point of the panoramic stitching in the rectangular bounding box; A pan / tilt platform connected to an imaging device to drive the imaging device to move; A capture movement trajectory acquisition unit, which is used to acquire the capture movement trajectory according to the starting point and the ending point of the panoramic stitching; A pan / tilt control unit connected to the pan / tilt and used to control the movement of the pan / tilt; the pan / tilt drives the imaging device to move, and when the rectangular field of view frame of the imaging device is at the starting position of the movement, after each movement along the capture movement trajectory, and at the end position of the movement, the image of the corresponding square is acquired through the imaging device connected to the pan / tilt; And an image stitching unit, which is connected to the imaging device and is used to stitch all the images acquired by the imaging device connected to the pan-tilt head according to an image panoramic stitching algorithm to obtain a panoramic stitching image.

Images