Image stitching method, electronic device and computer readable storage medium

Abstract

The present invention discloses an image stitching method, an electronic device and a computer-readable storage medium. The image stitching method comprises: obtaining a plurality of continuous frames of images to be stitched including a target object; extracting feature points of each frame of the images to be stitched respectively, and determining the feature points of each frame of the images to be stitched; matching the feature points between two adjacent frames of the images to be stitched that contain the most feature points, and using the successfully matched feature point pairs to determine the pixel movement distance between each adjacent image to be stitched; stitching each frame of the images to be stitched in sequence according to the time axis based on the pixel movement distance. Through the above manner, the present invention can improve the accuracy of image stitching and improve the reliability of the stitched image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image stitching method, an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of science and technology, images have been widely used in all walks of life. For example, image recognition, image extraction or image processing technologies have been gradually applied to all aspects of life.

[0003] When the image is taken by a fixed camera, it is generally difficult to capture the entire target object due to its fixed position and angle. For example, cameras at toll booths, vehicle entrances and exits, street monitoring, etc. can often only capture part of the target object. Therefore, when the target object needs to be processed, the images taken by the fixed camera need to be stitched to obtain the entire target object.

[0004] However, the current image stitching accuracy is not high, and stitched images are difficult to be widely used. Summary of the invention

[0005] The present invention provides an image stitching method, an electronic device and a computer-readable storage medium to solve the problem of low image stitching accuracy.

[0006] In order to solve the above technical problems, the present invention provides an image stitching method, comprising: acquiring a plurality of continuous frames of images to be stitched including a target object; extracting feature points of each frame of the images to be stitched respectively, and determining the feature points of each frame of the images to be stitched; matching the feature points between two adjacent frames of the images to be stitched containing the most feature points, and determining the pixel movement distance between each adjacent image to be stitched using the successfully matched feature point pairs; and stitching each frame of the images to be stitched in sequence according to the time axis based on the pixel movement distance.

[0007] The method of matching feature points between two adjacent frames of images to be spliced ​​that contain the most feature points includes: matching feature points between two adjacent frames of images to be spliced ​​that contain the most feature points among multiple frames of images to be spliced ​​to obtain multiple pairs of matching point pairs; and determining matching point pairs whose pixel movement meets preset requirements among the multiple pairs of matching point pairs as successfully matched feature point pairs, wherein the multiple frames of images to be spliced ​​are obtained by continuously shooting a moving target object at a preset frequency.

[0008] Among them, the matching point pairs whose pixel movement meets the preset requirements among multiple matching point pairs are determined as the successfully matched feature point pairs, including: in response to the pixel movement direction of the target matching point pair being the same as the movement direction of the target object, and the pixel movement distance of the target matching point pair being within a preset distance range, determining the target matching point pair as the successfully matched feature point pair.

[0009] The method of using the successfully matched feature point pairs to determine the pixel movement distance between adjacent images to be spliced ​​comprises: using the successfully matched feature point pairs to determine the pixel movement distance between two adjacent frames of images to be spliced ​​that contain the most feature points; and performing template matching on each adjacent image to be spliced ​​based on the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points to determine the pixel movement distance between each adjacent image to be spliced.

[0010] The method of using successfully matched feature point pairs to determine the pixel movement distance between two adjacent frames of images to be spliced ​​that contain the most feature points includes: determining the pixel distance between two feature points in each pair of successfully matched feature point pairs; comparing the pixel distances between each pair of successfully matched feature point pairs, and determining the pixel distance corresponding to the feature points with the same pixel distance and the largest number as the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points.

[0011] Among them, template matching is performed on each adjacent image to be spliced ​​based on the pixel movement distance between two adjacent frames of images to be spliced ​​containing the most feature points to determine the pixel movement distance between each adjacent image to be spliced, including: determining the matching range between each adjacent image to be spliced ​​based on the pixel movement distance between two adjacent frames of images to be spliced ​​containing the most feature points; and template matching is performed on each adjacent image to be spliced ​​within the matching range to respectively determine the pixel movement distance between each adjacent image to be spliced.

[0012] Among them, obtaining a continuous plurality of frames of images to be stitched includes: continuously photographing a moving target object at a preset frequency by a preset photographic device to obtain a continuous plurality of frames of original images; and performing orthographic projection correction on the plurality of frames of original images by using a transformation matrix to obtain a corresponding plurality of frames of images to be stitched.

[0013] Among them, the image stitching method also includes: obtaining a preset original image and its orthographic projection image, and determining four corner points on the preset original image, wherein the four corner points are rectangular corner points in the earth coordinate system; determining four corresponding matching points in the orthographic projection image based on the four corner points; and calculating the transformation matrix using the correspondence between the four corner points in the preset original image and the four matching points in the orthographic projection image.

[0014] The image stitching method further includes: determining a transformation matrix using a preset roll angle, a pitch angle and a yaw angle of the camera device.

[0015] The target object is a vehicle, and a plurality of continuous frames of images to be stitched including the target object are obtained, including: continuously photographing a moving vehicle at a preset frequency by a preset camera device to obtain a plurality of frames of images to be stitched; stitching each frame of the images to be stitched in sequence according to a timeline based on a pixel moving distance, including: stitching each frame of the images to be stitched in sequence according to a timeline based on a pixel moving distance to obtain a stitched image showing the entire vehicle.

[0016] To solve the above technical problems, the present invention further provides an electronic device, the electronic device comprising: a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement any of the above image stitching methods.

[0017] In order to solve the above technical problems, the present invention further provides a computer-readable storage medium, which stores program data. The program data can be executed to implement any of the above image stitching methods.

[0018] The beneficial effect of the present invention is as follows: Different from the prior art, the present invention first extracts feature points from each frame of the image to be spliced, determines the feature points of each frame of the image to be spliced, then matches the feature points between the two adjacent frames of the image to be spliced ​​that contain the most feature points, and uses the successfully matched feature point pairs to determine the pixel movement distance between each adjacent image to be spliced, and finally splices each frame of the image to be spliced ​​in the order of the time axis based on the pixel movement distance. Thus, the number of feature points in the two adjacent frames of the image to be spliced ​​that contain the most feature points can be used to increase the number of successfully matched feature point pairs, and then the precision and accuracy of determining the pixel movement distance between the two adjacent frames of the image to be spliced ​​that contain the most feature points through the successfully matched feature point pairs can be improved, and finally the reliability of the pixel movement distance between each adjacent image to be spliced ​​can be improved, the occurrence of splicing errors can be reduced, the image splicing precision can be improved, and the reliability of the spliced ​​image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of an embodiment of an image stitching method provided by the present invention;

[0020] Figure 2 is a flow chart of another embodiment of the image stitching method provided by the present invention;

[0021] Figure 3 is a structural schematic diagram of an embodiment of an electronic device provided by the present invention;

[0022] Figure 4 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the image stitching method provided by the present invention.

[0025] Step S11: acquiring a plurality of continuous frames of images to be stitched including the target object.

[0026] First, a plurality of continuous frames of images to be stitched including the target object are obtained, and image stitching is performed based on the plurality of frames of images to be stitched. Part of the target object can be sequentially displayed in the plurality of continuous frames of images to be stitched, so that the entire target object is obtained by stitching based on the partial target object in each frame of the images to be stitched.

[0027] In a specific implementation, this embodiment can be applied to an application scenario where the target object is moving, and the camera can only capture a portion of the target object. Multiple frames of images to be stitched are continuously captured for the moving target object, and the multiple frames of images to be stitched are stitched together to restore the entire target object. In another specific implementation, this embodiment can also be applied to an application scenario where the camera is moving, and the target object is in fixed or regular motion. Multiple frames of images to be stitched are continuously captured for the target object by a moving camera, and the multiple frames of images to be stitched are stitched together to restore the entire target object.

[0028] The continuous multiple frames of images to be stitched in this embodiment need to jointly show the entire target object or the preset position of the desired target object. Among them, the target object in this embodiment may include: vehicles, pedestrians, airplanes, animals, sky, ground and other objects, and the specific type of the specific target object is not limited here.

[0029] Step S12: extracting feature points of each frame of the image to be spliced, and determining the feature points of each frame of the image to be spliced.

[0030] After obtaining multiple consecutive frames of images to be stitched, feature points are extracted from each frame to be stitched, and feature points of each frame are determined. Feature points refer to points where the grayscale value of the image changes dramatically or points with large curvature on the edge of the image. Feature points can reflect the essential features of the image, so as to facilitate the identification of target objects in the image.

[0031] In a specific application scenario, feature points of each frame of the image to be spliced ​​can be extracted by using a DEM feature point extraction method, a Harris feature point extraction algorithm, a SIFT feature point extraction algorithm, a SURF feature point extraction algorithm or other feature point extraction algorithms to determine the feature points of each frame of the image to be spliced. The specific feature point extraction algorithm is not limited here. In another specific application scenario, a feature point extraction model can be pre-trained by first acquiring a training image and manually annotating feature points of the training image, so that feature points of each frame of the image to be spliced ​​can be extracted by using the trained feature point extraction model.

[0032] When extracting feature points from multiple consecutive images to be stitched of the same target object, the feature point extraction method for each image to be stitched is the same, so that the type of feature points of each image to be stitched is the same, which facilitates subsequent feature point matching.

[0033] Step S13: matching the feature points between two adjacent frames of images to be spliced ​​that contain the most feature points, and determining the pixel movement distance between each adjacent image to be spliced ​​using the successfully matched feature point pairs.

[0034] After obtaining the feature points of each frame of the image to be stitched, two adjacent frames of the image to be stitched containing the most feature points in the continuous multiple frames of the image to be stitched are determined, and the feature points between the two adjacent frames of the image to be stitched are matched to obtain a successfully matched feature point pair.

[0035] Since multiple frames of images to be stitched are continuous, the number of feature points in each of two adjacent frames of images to be stitched is gradual. By comparing the feature points in each of two adjacent frames of images to be stitched, the two adjacent frames of images to be stitched containing the most feature points are determined.

[0036] In a specific application scenario, when there are 10 consecutive frames of images to be stitched, and the numbers of feature points in the 10 frames of images to be stitched are 3, 4, 6, 7, 5, 4, 3, 3, 2, and 4 respectively, the two adjacent frames of images to be stitched containing the most feature points are the third frame and the fourth frame.

[0037] The feature points between two adjacent frames of images to be spliced ​​that contain the most feature points are matched to obtain a pair of successfully matched feature points.

[0038] Since the images to be stitched are continuous and for the same target object, the two adjacent frames of images to be stitched will show the same part of the target object, and the two adjacent frames of images to be stitched have some of the same features, and their feature points will be partially the same. By matching the feature points between the two adjacent frames of images to be stitched that contain the most feature points, the feature points showing the same part of the target object in the two adjacent frames of images to be stitched are determined, and the feature point pairs that are successfully matched are obtained.

[0039] If the successfully matched feature point pairs show the same part of the target object, the pixel movement distance between the two adjacent frames of images to be stitched that contain the most feature points is equal to the pixel movement distance between the two adjacent frames of images to be stitched that contain the most feature points. The pixel movement distance between each adjacent image to be stitched is determined by using the pixel movement distance between the two adjacent frames of images to be stitched that contain the most feature points.

[0040] In a specific application scenario, the step of determining the pixel movement distance between each adjacent image to be spliced ​​using the pixel movement distance between two adjacent frames of images to be spliced ​​containing the most feature points may include: calculating the pixel movement distance between each adjacent image to be spliced ​​based on the movement distance of the target object or the camera device and the pixel movement distance between two adjacent frames of images to be spliced ​​containing the most feature points. For example, when the target object moves at a constant speed, the pixel movement distance between the two adjacent frames of images to be spliced ​​containing the most feature points is the pixel movement distance between each adjacent image to be spliced, and when the speed of the target object or the camera device is other speeds, the pixel movement distance between each adjacent image to be spliced ​​can be calculated by using the specific speed and the pixel movement distance between the two adjacent frames of images to be spliced ​​containing the most feature points. In another specific application scenario, the step of determining the pixel movement distance between each adjacent image to be stitched using the pixel movement distance between two adjacent frames of images to be stitched that contain the most feature points may include: determining the matching distance between each adjacent image to be stitched based on the pixel movement distance between two adjacent frames of images to be stitched that contain the most feature points, matching each adjacent image to be stitched within the matching distance, and obtaining the pixel movement distance between each adjacent image to be stitched after the matching is successful.

[0041] Since the successfully matched feature point pairs are determined by the two adjacent frames of images to be spliced ​​that contain the most feature points, the number of their feature points can increase the number of successfully matched feature point pairs, thereby improving the precision and accuracy of the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points, and finally improving the reliability of the pixel movement distance between each adjacent image to be spliced, reducing the occurrence of splicing errors.

[0042] Step S14: splicing the frames of the images to be spliced ​​in sequence according to the time axis based on the pixel movement distance.

[0043] After obtaining the pixel movement distances between adjacent images to be stitched, the frames of images to be stitched are stitched in sequence according to the corresponding pixel movement distances in the order of the time axis, so that multiple frames of images to be stitched are stitched into a stitched image that can restore the entire target object.

[0044] Through the above steps, the image stitching method of this embodiment first extracts feature points from each frame of the image to be stitched, determines the feature points of each frame of the image to be stitched, then matches the feature points between the two adjacent frames of the image to be stitched that contain the most feature points, and uses the successfully matched feature point pairs to determine the pixel movement distance between each adjacent image to be stitched, and finally stitches each frame of the image to be stitched in the order of the time axis based on the pixel movement distance. In this way, the number of feature points in the two adjacent frames of the image to be stitched that contain the most feature points can be used to increase the number of successfully matched feature point pairs, and then the precision and accuracy of determining the pixel movement distance between the two adjacent frames of the image to be stitched that contain the most feature points through the successfully matched feature point pairs can be improved, and finally the reliability of the pixel movement distance between each adjacent image to be stitched is improved, the occurrence of stitching errors is reduced, and the image stitching accuracy is improved.

[0045] See also Figure 2 , Figure 2 It is a flowchart diagram of another embodiment of the image stitching method provided by the present invention.

[0046] Step S21: Continuously photograph the moving target object at a preset frequency using a preset camera device to obtain multiple frames of continuous original images, and perform orthographic projection correction on the multiple frames of original images using a transformation matrix to obtain multiple frames of images to be stitched.

[0047] The moving target object is continuously photographed at a preset frequency by a preset photographing device to obtain continuous multiple frames of original images. Among them, the preset photographing device of this embodiment is a photographing device with a fixed position and a fixed angle. The target object in the original image obtained by photographing the target object by the preset photographing device may have a certain tilt angle, so it is necessary to perform orthographic projection correction on the original image, so that the target object can face the photographing direction, which is convenient for subsequent image stitching. Among them, the preset photographing device may include a camera, a video camera, an intelligent terminal and other devices that can take images. The preset frequency can be set based on actual needs, for example, 2 frames per second, 5 frames per second, etc., which are not specifically limited. Among them, the setting of the preset frequency needs to enable the captured multiple frames of original images to show the entire target object or the preset position of the target object. Among them, the specific number of multiple frames of original images can be set according to the size of the target object and the preset frequency, which is not limited here.

[0048] The transformation matrix is ​​used to perform orthographic projection correction on multiple frames of original images respectively, and multiple frames of images to be stitched are obtained accordingly. The step of determining the transformation matrix of the preset camera device is performed before step S21. That is, the transformation matrix of the preset camera device is obtained in advance and directly used in the actual image stitching method.

[0049] In a specific implementation, a method for obtaining a transformation matrix may include:

[0050] First, a preset original image and its orthographic projection image are obtained. The preset original image is an original image captured by a preset camera device, and the orthographic projection image of the preset original image can be obtained by manual correction or deep learning model correction.

[0051] Four corner points are determined on the preset original image, where the four corner points are the corner points of a rectangle in the earth coordinate system. That is, the four corner points determined on the preset original image are four points that can form a rectangle in the real physical world. However, due to the shooting angle of the preset camera device, the four corner points may not form a rectangle on the preset original image. In a specific application scenario, an object with a rectangular surface or an object with a rectangular shape marked on it can be placed in the camera range of the preset camera device for shooting, to obtain a preset original image including the four corner points that can be determined.

[0052] Based on the four corner points in the preset original image, the corresponding four matching points are determined in the orthographic projection image. Since the orthographic projection image has been corrected for orthographic projection, its display direction is facing the preset camera device, so it can be determined that the four matching points form a rectangle in the orthographic projection image, that is, an orthographic projection is formed.

[0053] By calculating the correspondence between the four corner points in the preset original image and the four matching points in the orthographic projection image, a transformation matrix capable of performing orthographic projection correction on the original image can be obtained.

[0054] In a specific application scenario, the step of calculating the correspondence between four corner points in the preset original image and four matching points in the orthographic projection image may include:

[0055] Assuming that the coordinates of the four corner points in the preset original image are A(x1, y1), B(x2, y2), C(x3, y3), and D(x4, y4); the coordinates of the four matching points in the orthographic projection image are a(x′1, y′1), b(x′2, y′2), c(x′3, y′3), and d(x′4, y′4), the following formula can be obtained:

[0056]

[0057] in, is the transformation matrix, x, y are the coordinates of the pixel points on the preset original image, and x′1, y′1 are the coordinates of the corresponding pixel points on the orthographic projection image.

[0058] By substituting the coordinates of the four corner points and the four matching points into the following formulas:

[0059]

[0060] Then we can get 8 equations based on 8 unknowns, and then calculate h 11 、h 12 、h 13 、h 21 、h 22 、h 23 、h 31 、h 32 The solution of is obtained to obtain the specific parameters of the transformation matrix and determine the transformation matrix. Among them, since the degree of freedom of the matrix is ​​only 8, the present embodiment only needs to calculate 8 parameters to realize the orthographic projection correction of the original image.

[0061] In another specific application scenario, the correspondence between the four corner points in the preset original image and the four matching points in the orthographic projection image may also be calculated by other formulas, which are not limited here.

[0062] In another specific embodiment, the method for obtaining the transformation matrix may further include: determining the transformation matrix using the roll angle, pitch angle and yaw angle of the preset camera device. The roll angle is the deflection angle of the preset camera device relative to the X-axis direction, the pitch angle is the deflection angle of the preset camera device relative to the Y-axis direction, and the yaw angle is the deflection angle of the preset camera device relative to the Z-axis direction. The coordinate systems of the X-axis, Y-axis and Z-axis are established based on the movement direction of the target object and the movement direction of the preset camera device relative to the target object.

[0063] In a specific application scenario, the coordinate system can be established by taking the movement direction of the target object as the X-axis, the direction of the preset camera device perpendicular to the movement direction of the target object as the Y-axis, and the direction perpendicular to both the X-axis and the Y-axis as the Z-axis, thereby determining the roll angle, pitch angle and yaw angle of the preset camera device.

[0064] In a specific application scenario, a method for determining a transformation matrix based on a preset roll angle, pitch angle, and yaw angle of a camera device may include:

[0065] First determine the roll angle γ, pitch angle β and yaw angle α of the preset camera device, then the transformation matrix R c It can be calculated by the following formula:

[0066]

[0067] in,

[0068] In another specific application scenario, the method for determining the transformation matrix based on the roll angle, pitch angle, and yaw angle of the preset camera device may also be calculated by other formulas, which are not limited here.

[0069] After obtaining the transformation matrix, as long as the position and angle of the preset fixed camera do not change, the parameters of the transformation matrix do not need to be changed. Any original image taken by it can be obtained by calculating the coordinates of all pixel points in the original image using the transformation matrix, and arranging the pixel points based on the calculated coordinates to obtain the corresponding orthographic projection image.

[0070] Step S22: extracting feature points of each frame of the image to be stitched, and determining the feature points of each frame of the image to be stitched.

[0071] After obtaining multiple frames of images to be stitched after orthographic projection correction, feature points of each frame of the images to be stitched are extracted respectively to determine the feature points of each frame of the images to be stitched.

[0072] The specific steps of step S22 are the same as those of step S21 in the aforementioned embodiment. Please refer to the above text and will not be repeated here.

[0073] Step S23: matching the feature points between two adjacent frames of images to be spliced ​​that contain the most feature points among the multiple frames of images to be spliced ​​to obtain multiple pairs of matching point pairs; determining the matching point pairs whose pixel movement meets the preset requirements among the multiple pairs of matching point pairs as the successfully matched feature point pairs.

[0074] The feature points between two adjacent frames of images to be spliced ​​that contain the most feature points among the multiple frames of images to be spliced ​​are matched to obtain multiple pairs of matching point pairs. Among them, the matching point pairs are all the feature points that can be matched between two adjacent frames of images to be spliced. Among these matching point pairs, there may be feature point pairs on non-target objects. For example: when the preset camera device is set up on the gantry to shoot passing vehicles, each image will capture part of the fixed gantry, and the feature points of the fixed gantry cannot be used for image splicing. Therefore, this embodiment needs to screen the above multiple pairs of matching point pairs.

[0075] In a specific embodiment, the method for screening multiple pairs of matching point pairs may include: in response to the pixel movement direction of the target matching point pair being the same as the movement direction of the target object, and the pixel movement distance of the target matching point pair being within a preset distance range, determining that the target matching point pair is a successfully matched feature point pair. When the pixel movement direction of the target matching point pair is different from the movement direction of the target object or the pixel movement distance of the target matching point pair is not within the preset distance range, it is determined that the feature points in the target matching point pair are not the feature points corresponding to the moving target object, and then they are removed. Among them, the preset distance range can be set based on the moving speed of the target object, for example, 1-2 meters, 0.5-1 meters, etc., which are not specifically limited here. The preset requirement of this embodiment is that the pixel movement direction is the same as the moving direction of the target object, and the pixel movement distance is within the preset distance range. In other embodiments, the preset requirement can also be set according to the moving state of the target object, which is not limited here.

[0076] In another specific embodiment, the method for screening multiple pairs of matching point pairs may also include: screening multiple pairs of matching point pairs through a deep screening model trained by manual annotation or setting a screening area based on experience, etc., which is not specifically limited here.

[0077] The preset requirement of this step can filter out the fixed background feature points and debris feature points in the image to be stitched.

[0078] Step S24: using the successfully matched feature point pairs, determine the pixel movement distance between two adjacent frames of images to be spliced ​​that contain the most feature points.

[0079] After determining the successfully matched feature point pairs that meet the preset conditions, determine the pixel distance between the two feature points in each pair of successfully matched feature point pairs, and compare the pixel distances between each pair of successfully matched feature point pairs. The pixel distances corresponding to the feature points with the same pixel distance and the largest number of identical pixel distances are determined as the pixel movement distance between the two adjacent frames of images to be stitched that contain the most feature points.

[0080] Among them, the target object is generally a whole, and during the movement, the movement speed of each part is the same. If the successfully matched feature point pairs are feature points on the target object, the pixel movement distance of each pair of feature point pairs should be the same. Therefore, by comparing the pixel distance between each pair of successfully matched feature point pairs, and further screening the successfully matched feature point pairs using the pixel distance itself and the same number, it is possible to avoid interference from other moving objects or debris on the pixel movement distance, thereby improving the accuracy of the obtained pixel movement distance and enhancing the subsequent image stitching effect.

[0081] In a specific application scenario, when there are 5 pairs of successfully matched feature point pairs in two adjacent frames of images to be stitched that contain the most feature points, and the pixel distances between the two feature points in each pair of successfully matched feature points are 30, 20, 30, 30, and 20 pixels respectively, the pixel distance 30 with the same value and the largest number is used as the pixel movement distance between the two adjacent frames of images to be stitched that contain the most feature points.

[0082] In another specific application scenario, the RANSAC (Random Sampling Consensus) method can also be used to screen and iterate the successfully matched feature point pairs to determine the feature point pairs corresponding to the target object.

[0083] In this step, the pixel distance corresponding to the feature points with the same pixel distance and the largest number is determined as the pixel moving distance between two adjacent frames of images to be spliced ​​that contain the most feature points, so that the moving debris feature points in the images to be spliced ​​can be filtered.

[0084] Among them, in this step, to determine the pixel movement distance between two adjacent frames of images to be stitched that contain the most feature points, it is only necessary to calculate the pixel movement distance between the two frames of images to be stitched. Subsequently, matching is performed based on the pixel movement distance, which can reduce a large amount of matching calculations and improve stitching efficiency and speed.

[0085] Step S25: performing template matching on each adjacent image to be spliced ​​based on the pixel shift distance between two adjacent frames of images to be spliced ​​that contain the most feature points, so as to determine the pixel shift distance between each adjacent image to be spliced.

[0086] After obtaining the pixel movement distance between the two adjacent frames of images to be stitched that contain the most feature points, the pixel movement distance between each adjacent image to be stitched is determined, and the matching range between each adjacent image to be stitched is determined based on the pixel movement distance. Specifically, the error threshold can be added or subtracted from the pixel movement distance to obtain the matching range. The error threshold can be set based on the normal movement state of the target object, such as 5, 6, 7, etc. pixels, which are not specifically limited here.

[0087] In a specific application scenario, when the pixel movement distance is 30 pixels, the error threshold can be 5 pixels, and the matching range is 25-35 pixels. When template matching is performed on subsequent adjacent images to be spliced, template matching is performed within the movement range of 25-35 pixels.

[0088] Among them, template matching is to move one image to be stitched parallel to another image to be stitched. When the pixels on the two images overlap, the template matching is successful.

[0089] By performing template matching on every two adjacent images to be spliced ​​within the matching range, after every two adjacent images to be spliced ​​are successfully matched, the accurate pixel movement distance between every two adjacent images to be spliced ​​can be determined based on every two adjacent images to be spliced ​​that are successfully matched.

[0090] Step S26: splicing the frames of the images to be spliced ​​in sequence according to the time axis based on the pixel movement distance.

[0091] Based on the accurate pixel moving distance between each two adjacent images to be spliced, the frames of images to be spliced ​​are spliced ​​in order of the time axis, and the splicing seam area is fused to obtain a spliced ​​image. Specifically, the seam fusion can be performed by edge fusion technology or a splicing fuser.

[0092] In a specific application scenario, the image stitching method of this embodiment can be applied to vehicle image fusion, for example: a highway toll station takes pictures of passing vehicles to determine the toll standards, or a parking lot entrance and exit takes pictures of passing vehicles to determine the size of the vehicles.

[0093] First, a preset camera device with a fixed position and moving angle is used to continuously shoot a moving vehicle at a preset frequency to obtain multiple frames of images to be stitched. Feature points of each frame of the image to be stitched are extracted to determine the feature points of each frame of the image to be stitched. The feature points between two adjacent frames of the image to be stitched containing the most feature points are matched. The pixel movement distance between each adjacent image to be stitched is determined using the successfully matched feature point pairs. Finally, based on the pixel movement distance, each frame of the image to be stitched is stitched in sequence according to the time axis to obtain a stitched image showing the entire vehicle.

[0094] Among them, for a vehicle, the position with the most feature points is generally the front of the vehicle. Then, the feature points between two adjacent frames of images to be stitched containing the most feature points can be determined at the front of the vehicle for matching, thereby increasing the number of feature point pairs, and further improving the precision and accuracy of the pixel movement distance determined based on the feature point pairs.

[0095] Through the above steps, the image stitching method of this embodiment first continuously shoots the moving target object according to the preset frequency by the preset camera device to obtain continuous multiple frames of original images, and uses the transformation matrix to perform orthographic projection correction on the multiple frames of original images respectively, and correspondingly obtains multiple frames of images to be stitched, so that the target object on the image to be stitched is displayed positively, which is convenient for subsequent image stitching; then, feature points are extracted from each frame of the image to be stitched respectively, and the feature points of each frame of the image to be stitched are determined, and the feature points between two adjacent frames of the image to be stitched that contain the most feature points in the multiple frames of the image to be stitched are matched to obtain multiple pairs of matching point pairs; the matching point pairs whose pixel movement meets the preset requirements in the multiple pairs of matching point pairs are determined as the matching successful feature point pairs, so that the feature point pairs of the target object and the feature point pairs of the non-target object are screened by setting the preset requirements, and then the feature point pairs corresponding to the background, fixed structure or other sundries are filtered, so as to improve the accuracy and reliability of the subsequent calculation of the pixel movement distance based on the matching successful feature point pairs. This embodiment also first uses the successfully matched feature point pairs to determine the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points, and then performs template matching on each adjacent image to be spliced ​​based on the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points to determine the pixel movement distance between each adjacent image to be spliced, thereby performing double matching, that is, first performing a rough match based on the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points, and then performing a fine match based on the range of the rough match to accurately determine the pixel movement distance between each adjacent image to be spliced, while reducing the calculation cost, improving the accuracy and reliability of the pixel movement distance between each adjacent image to be spliced, and improving the accuracy of the subsequent spliced ​​image. Finally, based on the pixel movement distance, each frame of the image to be spliced ​​is spliced ​​in the order of the time axis to obtain an accurately spliced ​​spliced ​​image.

[0096] Based on the same inventive concept, the present invention also proposes an electronic device, which can be executed to implement the image stitching method of any of the above embodiments, see Figure 3 , Figure 3 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. The electronic device includes a processor 31 and a memory 32 .

[0097] The processor 31 is used to execute the program instructions stored in the memory 32 to implement the steps of any of the above-mentioned image stitching method embodiments. In a specific implementation scenario, the electronic device may include but is not limited to: a microcomputer, a server, and in addition, the electronic device may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0098] Specifically, the processor 31 is used to control itself and the memory 32 to implement the steps of any of the above-mentioned image stitching method embodiments. The processor 31 can also be called a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip with signal processing capabilities. The processor 31 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 31 can be implemented by an integrated circuit chip.

[0099] The above solution can improve the accuracy of image stitching and improve the reliability of the stitched image.

[0100] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, see Figure 4 , Figure 4 4 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. The computer-readable storage medium 40 stores at least one program data 41, and the program data 41 is used to implement any of the above methods. In one embodiment, the computer-readable storage medium 40 includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0101] In the several embodiments provided by the present invention, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0103] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0104] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium.

[0105] The above description is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image stitching method, characterized in that: The image stitching method comprises: A plurality of continuous frames of images to be stitched including the target object are obtained; wherein the moving target object is continuously photographed at a preset frequency by a preset camera device at a fixed position and a fixed angle to obtain a plurality of continuous frames of original images; and a transformation matrix is ​​used to perform orthographic projection correction on the plurality of frames of the original images respectively to obtain a plurality of frames of the images to be stitched correspondingly; Extracting feature points of the images to be stitched in each frame respectively to determine the feature points of the images to be stitched in each frame; Matching feature points between two adjacent frames of images to be stitched that contain the most feature points, and using the successfully matched feature point pairs to determine the pixel distance between the two adjacent frames of images to be stitched that contain the most feature points, and calculating the pixel movement distance between each adjacent image to be stitched based on the pixel distance between the two adjacent frames of images to be stitched that contain the most feature points and the speed of the target object; Based on the pixel moving distance, the frames of the to-be-joined images are joined in sequence along the time axis.

2. The image stitching method according to claim 1, characterized in that: The matching of feature points between two adjacent frames of images to be spliced ​​that contain the most feature points includes: Matching the feature points between two adjacent frames of images to be stitched that contain the most feature points among the multiple frames of images to be stitched, to obtain multiple pairs of matching point pairs; The matching point pairs whose pixel movement meets the preset requirements among the multiple matching point pairs are determined as the successfully matched feature point pairs, wherein the multiple frames of images to be stitched are obtained by continuously shooting the moving target object at a preset frequency.

3. The image stitching method according to claim 2, characterized in that: The step of determining the matching point pairs whose pixel movement meets the preset requirement among the plurality of matching point pairs as the successfully matched feature point pairs comprises: In response to the pixel movement direction of the target matching point pair being the same as the movement direction of the target object, and the pixel movement distance of the target matching point pair being within a preset distance range, the target matching point pair is determined to be the successfully matched feature point pair.

4. The image stitching method according to claim 2 or 3, characterized in that: The method of using the successfully matched feature point pairs to determine the pixel movement distance between adjacent images to be stitched includes: The pixel moving distance between two adjacent frames of images to be spliced ​​containing the most feature points is determined by using the successfully matched feature point pairs; Template matching is performed on each adjacent image to be spliced ​​based on the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points, so as to determine the pixel movement distance between each adjacent image to be spliced.

5. The image stitching method according to claim 4, characterized in that: The method of using the successfully matched feature point pairs to determine the pixel movement distance between two adjacent frames of images to be spliced ​​that contain the most feature points includes: Determine the pixel distance between the two feature points in each pair of successfully matched feature points; The pixel distances between each pair of successfully matched feature points are compared, and the pixel distances corresponding to the feature points with the same pixel distances and the largest number of the same pixel distances are determined as the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points.

6. The image stitching method according to claim 4, characterized in that: The performing template matching on each adjacent image to be spliced ​​based on the pixel movement distance between the two adjacent frames of images to be spliced ​​that contain the most feature points to determine the pixel movement distance between each adjacent image to be spliced ​​includes: Determine the matching range between adjacent images to be stitched based on the pixel moving distance between two adjacent frames of images to be stitched that contain the most feature points; Template matching is performed on each adjacent image to be spliced ​​within the matching range to determine the pixel movement distance between each adjacent image to be spliced.

7. The image stitching method according to claim 1, characterized in that: The image stitching method further comprises: Obtain a preset original image and its orthographic projection image, and determine four corner points on the preset original image, wherein the four corner points are rectangular corner points in the earth coordinate system; Determining corresponding four matching points in the orthographic projection image based on the four corner points; The transformation matrix is ​​calculated using the correspondence between the four corner points in the preset original image and the four matching points in the orthographic projection image.

8. The image stitching method according to claim 1, characterized in that: The image stitching method further comprises: The transformation matrix is ​​determined using the roll angle, pitch angle and yaw angle of the preset camera device.

9. The image stitching method according to claim 1, characterized in that: The target object is a vehicle, The step of obtaining a plurality of continuous frames of images to be stitched including the target object comprises: Continuously photographing the moving vehicle at a preset frequency using a preset camera device to obtain the multiple frames of images to be stitched; The step of sequentially splicing the frames of the images to be spliced ​​in the order of the time axis based on the pixel moving distance includes: Based on the pixel movement distance, the frames of the images to be spliced ​​are sequentially spliced ​​in the order of the time axis to obtain a spliced ​​image showing the entire vehicle.

10. An electronic device, characterized in that: The electronic device comprises: a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the image stitching method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and the program data can be executed to implement the image stitching method according to any one of claims 1 to 9.

Images