A video picture splicing method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a video picture splicing method, device, equipment and storage medium, a video picture to be spliced shot by a camera is acquired; image feature points are identified from the video picture to be spliced; according to the image feature points, space feature points corresponding to the image feature points are matched; a feature point pair composed of the image feature points and the space feature points is screened to obtain a target feature point pair; according to the target feature point pair, extrinsic parameters of the camera are calculated; according to the extrinsic parameters of the camera, the video picture to be spliced is spliced to obtain a target video picture after splicing. Embodiments of the present application identify image feature points from the video picture to be spliced, match corresponding space feature points, screen the feature point pair composed of the image feature points and the space feature points, and filter out coarse noise points generated when the matching feature points are extracted, thereby improving the accuracy of camera extrinsic parameter calculation in a harsh environment and ensuring the success rate and precision of subsequent video picture splicing.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video splicing method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] Achieving unattended production environments in hazardous settings such as underground mines is a significant development direction in the mining industry. By introducing advanced automation technologies and artificial intelligence, the working status of coal mining machines can be monitored in real time, significantly reducing the risk of personal injury to miners underground, thereby improving working conditions and saving human resources.

[0003] In recent years, monocular cameras have attracted increasing attention due to their ease of installation and low cost. Monocular camera video stitching technology utilizes a multi-camera system to capture images from different perspectives and automatically stitches them into one or more wide-field-of-view or panoramic images using specific image algorithms. This method not only significantly reduces equipment costs but also makes it suitable for more flexible application scenarios.

[0004] Despite advancements in monocular camera-based video stitching technology, challenges remain in real-time performance, stitching accuracy, and handling complex environmental changes. In coal mine monitoring applications, coal dust generated during mining operations can interfere with camera footage, leading to numerous bad pixels when extracting feature points from the machine for matching algorithms. This affects the accuracy of camera extrinsic parameter calculations, potentially resulting in stitching failures or noticeable stitching artifacts. Summary of the Invention

[0005] This application provides a video splicing method to solve or at least partially solve the above-mentioned problems.

[0006] Accordingly, embodiments of this application also provide a video splicing device, an electronic device, and a storage medium to ensure the implementation and application of the above method.

[0007] To address the aforementioned problems, this application discloses a video frame stitching method, the method comprising:

[0008] Acquire video footage captured by the camera to be stitched together;

[0009] Identify image feature points from the video frames to be stitched together;

[0010] Based on the image feature points, match the spatial feature points corresponding to the image feature points;

[0011] By filtering the feature point pairs formed by the image feature points and the spatial feature points, the target feature point pairs are obtained;

[0012] Calculate the extrinsic parameters of the camera based on the target feature point pairs;

[0013] Based on the external parameters of the camera, the video frames to be stitched together are spliced ​​to obtain the spliced ​​target video frame.

[0014] In one optional embodiment of this application, identifying image feature points from the video frames to be stitched includes:

[0015] The video frames to be stitched are subjected to distortion correction processing to obtain the distortion-corrected video frames;

[0016] A recognition model is used to identify the image features in the distortion-free video frame; the recognition model is a model trained using training images containing the image features.

[0017] Based on the image features, the image feature points are obtained.

[0018] In an optional embodiment of this application, obtaining the image feature points based on the image features includes:

[0019] Based on the image features, the distortion-free video image is cropped to obtain a feature thumbnail that includes the image features;

[0020] Extract the image feature points from the feature thumbnail;

[0021] The step of matching spatial feature points corresponding to the image feature points includes:

[0022] Obtain the spatial feature points corresponding to the image features;

[0023] Obtain a reference image containing reference feature points; the coordinate system of the reference image and the spatial coordinate system have a first pairing relationship, and the spatial coordinate system is the coordinate system where the spatial feature points are located;

[0024] Match the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point;

[0025] The spatial feature points are matched based on the first pairing relationship and the second pairing relationship.

[0026] In an optional embodiment of this application, the step of filtering the feature point pairs composed of the image feature points and the spatial feature points to obtain target feature point pairs includes:

[0027] Set the number of iterations;

[0028] Based on the number of iterations, a random sample consensus algorithm is used to filter out noise point pairs in the feature point pairs to obtain the target feature point pairs.

[0029] In an optional embodiment of this application, after calculating the extrinsic parameters of the camera based on the target feature point pairs, the method further includes:

[0030] The target feature point pairs are classified to obtain a first feature point pair, a second feature point pair, and a third feature point pair;

[0031] The first reprojection error is calculated based on the first feature point pair, the second feature point pair, and the third feature point pair.

[0032] If the first projection error is greater than a preset error threshold, then the second projection error is calculated based on the first feature point pair and the second feature point pair.

[0033] If the second projection error is greater than the preset error threshold, then the third projection error is calculated based on the first feature point pair.

[0034] If the third projection error is greater than the preset error threshold, the extrinsic parameters of the camera are corrected to the extrinsic parameters calculated based on the previous video frame.

[0035] In one optional embodiment of this application, the extrinsic parameters of the camera include a rotation matrix and a translation matrix. The step of stitching the video frames to be stitched together based on the extrinsic parameters of the camera to obtain the stitched target video frame includes:

[0036] Using the rotation matrix and the translation matrix, an affine transformation is performed on the distortion-free video frame to obtain the transformed video frame;

[0037] The overlapping area of ​​the transformed video frame is obtained based on the translation matrix.

[0038] Based on the overlapping area, the transformed video frames are stitched together to obtain the stitched target video frame.

[0039] In one optional embodiment of this application, the method is applied to a coal mine monitoring system. The video footage to be stitched includes a coal mining machine, which includes a body structure and an identification card on the machine body. The identification card has a unique number. Identifying image feature points from the video footage to be stitched includes:

[0040] Identify the fuselage structure and the number on the identification card from the video footage to be stitched together;

[0041] Obtain the position coordinates of the fuselage structure and the identification card in the video frame to be stitched together;

[0042] Based on the location coordinates, the image feature points are identified from the video frames to be stitched together.

[0043] This application also discloses a video splicing device, the device comprising:

[0044] The video capture module is used to acquire video footage captured by the camera for stitching.

[0045] An image feature point recognition module is used to identify image feature points from the video frames to be stitched together.

[0046] The spatial feature point calculation module is used to match the spatial feature points corresponding to the image feature points based on the image feature points.

[0047] The feature point pair filtering module is used to filter feature point pairs composed of the image feature points and the spatial feature points to obtain target feature point pairs.

[0048] The extrinsic parameter calculation module is used to calculate the extrinsic parameters of the camera based on the target feature point pairs;

[0049] The video splicing module is used to splice the video frames to be spliced ​​according to the external parameters of the camera to obtain the spliced ​​target video frame.

[0050] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, wherein when the executable code is executed, the processor performs one or more video frame stitching methods as described in this application.

[0051] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform one or more video frame stitching methods as described in this application.

[0052] Compared with the prior art, the embodiments of this application have the following advantages:

[0053] In this embodiment, a video frame to be stitched is acquired from a camera; image feature points are identified from the video frame; spatial feature points corresponding to the image feature points are matched; feature point pairs composed of image feature points and spatial feature points are filtered to obtain target feature point pairs; the camera's extrinsic parameters are calculated based on the target feature point pairs; and the video frame to be stitched is stitched together based on the camera's extrinsic parameters to obtain the stitched target video frame. This embodiment improves the accuracy of camera extrinsic parameter calculation in harsh environments by identifying image feature points in the video frame to be stitched, matching corresponding spatial feature points, and filtering feature point pairs composed of image feature points and spatial feature points. This removes coarse noise points generated during feature point extraction and matching, ensuring the success rate and accuracy of subsequent video frame stitching and effectively avoiding stitching failures or obvious stitching artifacts. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the steps of an embodiment of a video frame splicing method according to this application;

[0055] Figure 2 This is a flowchart of feature point matching according to an embodiment of a video frame stitching method of this application;

[0056] Figure 3 This is a flowchart illustrating the feature point pair acquisition process of one embodiment of the video frame stitching method of this application;

[0057] Figure 4 This is a flowchart of the external parameter optimization of an embodiment of the video frame stitching method of this application;

[0058] Figure 5 This is a detailed flowchart of an embodiment of a video frame splicing method according to this application;

[0059] Figure 6 This is a structural block diagram of an embodiment of a video splicing device according to this application;

[0060] Figure 7 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0061] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Reference Figure 1 This is a flowchart illustrating the steps of an embodiment of a video frame stitching method according to this application, including the following steps:

[0063] Step 101: Obtain the video footage captured by the camera to be stitched together.

[0064] In this embodiment of the application, before step 101, multiple monocular cameras can be placed at fixed intervals in the area where monitoring and shooting are required, and the distance from all cameras to the device to be detected is equal, and the straight-line distance from the camera to the device is recorded.

[0065] In one embodiment, the area to be monitored and filmed can be a coal mine, and the equipment to be monitored can be a coal mining machine. The coal mining machine can be divided into three structural parts: a primary main body, a secondary body capable of vertically swinging, and a drum cutting section.

[0066] In step 101, video frames to be stitched are acquired from the current frames captured by multiple cameras.

[0067] Step 102: Identify image feature points from the video frames to be stitched together.

[0068] In one optional embodiment, identification cards can be affixed to the surface of the device to be tested in advance, and the spatial coordinates of the four corner points of the identification cards and the various body structures of the device in the spatial coordinate system can be recorded as spatial feature points. The affixed identification cards include directly identifiable numbers and the shape combination of the identification cards is unique.

[0069] Furthermore, image data of the identification card and various body structures of the device can be collected from multiple angles. After labeling the image coordinate data, a suitable neural network is selected and trained based on the image data to obtain an image recognition model. The image recognition model can identify the identification card and various body structures of the device from the video footage.

[0070] In step 102, an image recognition model can be used to identify relevant image features from the video frames to be stitched together. Based on the identified image features, the video frames to be stitched are cropped to obtain feature thumbnails. Then, feature corner points are extracted from the feature thumbnails as image feature points for that frame. In one embodiment, the image features may include the coordinates and corresponding number of the identification card, and various body structures of the device.

[0071] Step 103: Calculate the spatial feature points corresponding to the image feature points based on the image feature points.

[0072] In step 103, based on the image feature points, the corresponding spatial feature points are matched to obtain 2D-3D feature point pairs. The spatial feature points are obtained in advance by measuring the spatial coordinates of the actual objects corresponding to the image features.

[0073] Step 104: Filter the feature point pairs formed by the image feature points and the spatial feature points to obtain the target feature point pairs.

[0074] In step 104, target feature point pairs are selected from feature point pairs composed of image feature points and spatial feature points. Specifically, the number of iterations can be set to perform RANSAC (Random Sample Consensus) sampling calculations to select interior points and filter out exterior points with high noise, thereby obtaining target feature point pairs.

[0075] In one embodiment, if the obtained target feature point pairs are less than 4 pairs, the extrinsic parameters of the frame are not updated. That is, the extrinsic parameters of the frame are not calculated using the target feature point pairs of the frame, but the extrinsic parameters calculated using the previous frame of the camera are used instead.

[0076] Step 105: Calculate the extrinsic parameters of the camera based on the target feature point pairs.

[0077] In step 105, the extrinsic parameters of the camera are calculated based on the target feature point pairs, that is, the mapping relationship between the coordinates of the feature points in the world coordinate system and the coordinates in the image coordinate system is established.

[0078] Specifically, in one optional embodiment, the camera extrinsic parameters can be calculated using the PNP method (Perspective-n-Point, a method for calculating camera pose from multiple points in three-dimensional space). From the perspective projection model, according to formula (1), the pixel coordinate system (O) of the image can be obtained. uv uv) and camera coordinate system (O) c X c Y c Z c The relationship between )

[0079]

[0080] Where u and v are the image coordinates of the image feature points; K is the camera's intrinsic parameter matrix; X c Y c Z c These are the coordinates in the camera coordinate system.

[0081] Therefore, according to formula (2), the three-dimensional coordinates of the imaging point on the normalized image plane can be obtained:

[0082]

[0083] Where X1, Y1, and Z1 are the three-dimensional coordinates of the imaging points on the normalized image plane; u and v are the image coordinates of the image feature points; and K is the camera's intrinsic parameter matrix.

[0084] Then, based on the perspective projection model, and according to the pixel coordinate system (O) of the image in formula (2), uv The relationship between uv) and the normalized image plane coordinate system is further obtained according to formula (3) to obtain the camera coordinate system (O).c X c Y c Z c ) and world coordinate system (O) w X w Y w Z w The relationship between )

[0085]

[0086] Where R and t are the extrinsic parameters of the camera, R is the rotation matrix and T is the translation matrix; X c Y c Z c The coordinates in the camera coordinate system; X w Y w Z w These are the coordinates in the world coordinate system, i.e., the coordinates of spatial feature points.

[0087] Thus, given the camera's intrinsic parameters, the extrinsic rotation and translation matrices for each monocular camera in the current frame can be calculated using target feature point pairs.

[0088] Step 106: Based on the external parameters of the camera, stitch together the video frames to be stitched to obtain the stitched target video frames.

[0089] In step 106, based on the calculated extrinsic parameters of the cameras, the video frames to be stitched from the current frames acquired by several cameras are stitched together to obtain the stitched target video frame.

[0090] In this embodiment, a video frame to be stitched is acquired from a camera; image feature points are identified from the video frame; spatial feature points corresponding to the image feature points are matched; feature point pairs composed of image feature points and spatial feature points are filtered to obtain target feature point pairs; the camera's extrinsic parameters are calculated based on the target feature point pairs; and the video frame to be stitched is stitched together based on the camera's extrinsic parameters to obtain the stitched target video frame. This embodiment improves the accuracy of camera extrinsic parameter calculation in harsh environments by identifying image feature points in the video frame to be stitched, matching corresponding spatial feature points, and filtering feature point pairs composed of image feature points and spatial feature points. This removes coarse noise points generated during feature point extraction and matching, ensuring the success rate and accuracy of subsequent video frame stitching and effectively avoiding stitching failures or obvious stitching artifacts.

[0091] In one optional embodiment of this application, step 102 includes:

[0092] The video frames to be stitched are subjected to distortion correction processing to obtain the distortion-corrected video frames;

[0093] A recognition model is used to identify the image features in the distortion-free video frame; the recognition model is a model trained using training images containing the image features.

[0094] Based on the image features, the image feature points are obtained.

[0095] In this embodiment, before identifying the image features in the video frame, the spliced ​​video frame needs to be distorted to obtain a distorted video frame. Then, a recognition model is used to identify the image features in the distorted video frame. The recognition model is a model trained using training images containing image features. In one embodiment, the image features may include the location coordinates and corresponding number of the identification card of the coal mining machine in the coal mine, and the various body structures of the equipment.

[0096] Furthermore, image feature points can be extracted from the video frame based on the identified image features after distortion correction.

[0097] In one embodiment, image data of the identification card and various body structures of the device, including images taken from multiple angles, can be collected. After labeling the image coordinate data, a suitable neural network is selected, and the neural network is trained based on the image data to obtain an image recognition model.

[0098] In another embodiment, if the device to be tested is a coal mining machine, since the drum cutting part is generally not visible when the coal mining machine is working, the primary main body and the secondary swingable body of the coal mining machine can be used as the device body structure in the image data for training.

[0099] In another embodiment, to ensure the real-time performance of video frame stitching, the YOLO (You Only Look Once, an object detection algorithm) detection method, which has fast detection speed and strong real-time performance, can be used to identify the image features in the distortion-free video frame.

[0100] The embodiments of this application can use a recognition model to identify the image features in the distortion-free video frame, thereby obtaining image feature points, which can improve the recognition accuracy of image feature points, and thus be used for subsequent calculation of camera extrinsic parameters.

[0101] In an optional embodiment of this application, obtaining the image feature points based on the image features includes:

[0102] Based on the image features, the distortion-free video image is cropped to obtain a feature thumbnail that includes the image features;

[0103] Extract the image feature points from the feature thumbnail;

[0104] Step 103 includes:

[0105] Obtain the spatial feature points corresponding to the image features;

[0106] Obtain a reference image containing reference feature points; the coordinate system of the reference image and the spatial coordinate system have a first pairing relationship, and the spatial coordinate system is the coordinate system where the spatial feature points are located;

[0107] Match the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point;

[0108] The spatial feature points are matched based on the first pairing relationship and the second pairing relationship.

[0109] In this embodiment, image feature points are extracted from the distortion-free video frame, and corresponding spatial feature points are matched based on the image feature points. Specifically, this may include the following steps.

[0110] When applied to a coal mine monitoring system in this embodiment, due to the low visibility underground, the presence of numerous similar structures in the image, and significant interference from coal dust, the method of extracting feature corner points from the entire image and matching them with reference feature points introduces serious mismatch problems, greatly affecting subsequent camera extrinsic parameter calculations and video image stitching. Therefore, it is possible to first crop the distorted video image based on the image features to obtain a feature minimap including the image features, and then extract the feature corner points of the feature minimap as image feature points.

[0111] Obtain the spatial feature points corresponding to the physical objects of the pre-measured image features, and a reference image containing the reference feature points. There is a first pairing relationship between the coordinate system of the reference image and the spatial coordinate system, where the spatial coordinate system is the coordinate system where the spatial feature points are located. Specifically, in an optional embodiment, the size transformation relationship between the pixel points and the actual length in space can be calculated in advance according to formula (4), resulting in the following size transformation relationship:

[0112]

[0113] Where FOV is the field of view of a monocular camera; L is the maximum world coordinate system length that the camera can capture; and d is the distance from the camera to the device.

[0114] Therefore, the size transformation between the world coordinate system and the image plane can be obtained according to formula (5), that is, the length (cm, centimeters) of a pixel in the image corresponding to the world coordinate system:

[0115]

[0116] Wherein, the resolution of the monocular camera image is W×H pixels; FOV is the field of view of the monocular camera; and d is the distance from the camera to the device.

[0117] Thus, based on the relationship between the length of a pixel in the image and the length of the corresponding point in the world coordinate system, the coordinate information of the reference feature point on the reference image corresponding to the actual device feature point in the world coordinate system can be obtained, that is, the first pairing relationship between the coordinate system of the reference image and the spatial coordinate system.

[0118] A brute-force matching method is used to match image feature points of a feature patch with the reference feature point, obtaining a second pairing relationship between the image feature point and the reference feature point, i.e., a 2D-2D feature point pair. Therefore, based on the first and second pairing relationships, the corresponding spatial feature points of the image feature points can be matched, i.e., a 2D-3D feature point pair.

[0119] Reference Figure 2 This is a flowchart illustrating the feature point matching process of one embodiment of a video splicing method according to this application.

[0120] like Figure 2 As shown, when numbered identification cards are affixed to the surface of the device to be inspected, training images of identification card groups at different angles can be used to pre-train the image recognition model. The labels of the training images are the identification card numbers, thus obtaining the trained image recognition model.

[0121] The system acquires the current frame from a single-channel camera after distortion correction. A trained image recognition model is used to detect the identification card numbers and their corresponding coordinates as image features. Based on the detected card coordinates, a feature map is cropped from the video frame by a factor of 1.1. The coordinates of the feature corners are extracted from this feature map as image feature points. Reference feature point coordinates from a reference image are also extracted. A brute-force matching method is used to obtain a second pairing relationship between the image feature points and the reference feature points, i.e., 2D-2D feature point pairs.

[0122] Figure 2 The specific values ​​for feature image cropping are merely examples; those skilled in the art can set the corresponding values ​​according to actual needs.

[0123] Reference Figure 3 This is a flowchart illustrating the feature point pair acquisition process of an embodiment of a video splicing method according to this application.

[0124] like Figure 3As shown, the second pairing relationship between the feature corner points of the feature thumbnail cropped from the camera video frame and the reference feature points in the reference image (i.e., 2D-2D feature point pairs), and the first pairing relationship between the reference feature points on the reference image and the actual device feature points in the world coordinate system (i.e., the true coordinate information of the feature points on the identification card in the world coordinate system) obtained by the relationship between the length of a pixel in the image and the length of the pixel in the world coordinate system, constitute a 2D-3D correspondence. Furthermore, based on the first and second pairing relationships, the spatial feature points corresponding to the image feature points can be matched, i.e., 2D-3D feature point pairs corresponding to the image coordinate system and the spatial coordinate system.

[0125] This application embodiment extracts image feature points by cropping feature thumbnails, eliminating the need to extract image feature points from the entire video frame. This reduces resource consumption during image feature point extraction and improves the extraction speed. By matching feature corner points of feature thumbnails with reference feature points, mismatch problems that may occur when directly matching feature corner points of the entire image with reference feature points can be avoided, resulting in more accurate feature point pairs for subsequent calculation of camera extrinsic parameters.

[0126] In one optional embodiment of this application, step 104 includes:

[0127] Set the number of iterations;

[0128] Based on the number of iterations, a random sample consensus algorithm is used to filter out noise point pairs in the feature point pairs to obtain the target feature point pairs.

[0129] In this embodiment, a random sample consensus algorithm can be performed to calculate the number of iterations and filter out noisy point pairs in the feature point pairs to obtain the target feature point pairs. Specifically, it is possible to determine whether a feature point is an interior point, i.e., a feature point in the dataset that conforms to the fitted model, based on formula (6), thereby selecting the target feature point pairs.

[0130]

[0131] Where Inlier represents an interior point; Outlier represents an exterior point; t is a threshold value, chosen such that the probability of the feature point being an interior point is α, typically α = 95%, and t... 2 The values ​​can be determined by reading the table; F and δ are preset parameters; d is the distance from the feature point to the fitted model.

[0132] The embodiments of this application can further filter the obtained feature point pairs, thereby filtering out noise point pairs in the feature point pairs and ensuring the accuracy of camera extrinsic parameter calculation.

[0133] In one optional embodiment of this application, after step 105, the method further includes:

[0134] The target feature point pairs are classified to obtain a first feature point pair, a second feature point pair, and a third feature point pair;

[0135] The first reprojection error is calculated based on the first feature point pair, the second feature point pair, and the third feature point pair.

[0136] If the first projection error is greater than a preset error threshold, then the second projection error is calculated based on the first feature point pair and the second feature point pair.

[0137] If the second projection error is greater than the preset error threshold, then the third projection error is calculated based on the first feature point pair.

[0138] If the third projection error is greater than the preset error threshold, the extrinsic parameters of the camera are corrected to the extrinsic parameters calculated based on the previous video frame.

[0139] In this embodiment, feature point pairs can be further classified by type according to the objective function of minimizing reprojection error, and the extrinsic parameters can be optimized step by step to improve the accuracy of extrinsic parameter calculation.

[0140] The target feature point pairs are classified to obtain a first feature point pair, a second feature point pair, and a third feature point pair. In an optional embodiment, feature point pairs with clear card surfaces and high recognition accuracy can be used as the first feature point pair; feature point pairs with lower recognition accuracy of the primary body but minimal change during device operation can be used as the second feature point pair; and feature point pairs with lower recognition accuracy of the secondary body and changes in the vertical swing of the body during device operation can be used as the third feature point pair.

[0141] Reference Figure 4 This is a flowchart of the external parameter optimization of an embodiment of a video splicing method of this application.

[0142] In this embodiment, the objective function for minimizing reprojection error aims to minimize the error between the projection of spatial feature points onto the image and the actual image feature points by optimizing the extrinsic parameters of the camera.

[0143] like Figure 4 As shown, the first extrinsic parameters of the camera are first calculated based on the first feature point pair, the second feature point pair, and the third feature point pair. Then, the first reprojection error is calculated using the PNP method based on the first extrinsic parameters. That is, the projection of the spatial feature points on the image is calculated using the first extrinsic parameters, and the error between the projection of the spatial feature points on the image and the actual image feature points is calculated.

[0144] If the first projection error is greater than the preset error threshold, the second extrinsic parameter of the camera is calculated based on the first feature point pair and the second feature point pair. The second projection error is calculated using the PNP method based on the second extrinsic parameter. That is, the projection of the spatial feature point on the image is calculated through the second extrinsic parameter, and the error between the projection of the spatial feature point on the image and the actual image feature point is calculated.

[0145] If the second projection error is greater than the preset error threshold, the third extrinsic parameter of the camera is calculated based on the first feature point pair, and the third projection error is calculated using the PNP method based on the third extrinsic parameter. That is, the projection of the spatial feature point on the image is calculated through the third extrinsic parameter, and the error between the projection of the spatial feature point on the image and the actual image feature point is calculated.

[0146] If the third projection error is still greater than the preset error threshold, the target feature points of this frame will not be used to update the camera's extrinsic parameters R / T (rotation matrix / translation matrix). Instead, the camera's extrinsic parameters will be corrected to the extrinsic parameters calculated based on the previous video frame.

[0147] If the first projection error is less than the preset error threshold, or the second projection error is less than the preset error threshold, or the third projection error is less than the preset error threshold, then the target feature points of this frame can be used to update the camera's extrinsic parameters R / T.

[0148] The embodiments of this application can optimize the calculation of camera extrinsic parameters step by step according to the reprojection error, so as to ensure the accuracy of the calculated camera extrinsic parameters, thereby ensuring the success rate and accuracy of subsequent video splicing, and effectively avoiding splicing failure or obvious splicing marks.

[0149] In an optional embodiment of this application, the extrinsic parameters of the camera include a rotation matrix and a translation matrix, and step 106 includes:

[0150] Using the rotation matrix and the translation matrix, an affine transformation is performed on the distortion-free video frame to obtain the transformed video frame;

[0151] The overlapping area of ​​the transformed video frame is obtained based on the translation matrix.

[0152] Based on the overlapping area, the transformed video frames are stitched together to obtain the stitched target video frame.

[0153] In this embodiment, the distortion-free video footage can be stitched together based on the camera's extrinsic parameters to obtain the stitched target video footage.

[0154] Specifically, a rotation matrix and a translation matrix are used to perform an affine transformation on the distortion-free video image to obtain the transformed video image. In an optional embodiment, since the device is often placed in the center of the shooting frame, the transformed video image can be cropped or have black borders added as needed.

[0155] The translation matrix reveals the offset distance of image feature points from each camera relative to the origin. This allows us to obtain the overlapping area of ​​two consecutive video frames in the processed channel. Within this overlapping area, a weighted average method is used to smooth the image transition, thus completing the stitching of video frames from multiple cameras and obtaining the stitched target video frame.

[0156] The embodiments of this application can stitch together the distortion-free video footage according to the camera's extrinsic parameters to obtain the stitched target video footage, thus ensuring the success rate and accuracy of video footage stitching.

[0157] In one optional embodiment of this application, applied to a coal mine monitoring system, the video footage to be stitched includes a coal mining machine, the coal mining machine includes a body structure and an identification card on the body, the identification card having a unique number, and step 102 includes:

[0158] Identify the fuselage structure and the number on the identification card from the video footage to be stitched together;

[0159] Obtain the position coordinates of the fuselage structure and the identification card in the video frame to be stitched together;

[0160] Based on the location coordinates, the image feature points are identified from the video frames to be stitched together.

[0161] In this embodiment, if the video stitching method shown in this embodiment is applied to a coal mine monitoring system, the area to be monitored and filmed can be a coal mine, and the equipment to be monitored can be a coal mining machine. The coal mining machine can be divided into three structural parts: a primary main body, a secondary body that can swing up and down, and a drum cutting section. Identification cards can be pre-attached to the surface of the coal mining machine. The affixed identification cards include directly identifiable numbers, and the shape combination of the identification cards is unique.

[0162] Therefore, in this embodiment, when acquiring image feature points, the image features of the fuselage structure and the identification card in the video frame to be stitched can be acquired, and feature small images can be cropped according to the image features. The feature corner points in the feature small images are used as image feature points, and their position coordinates are used as image feature points for subsequent video stitching steps.

[0163] The embodiments of this application can be applied to coal mine areas with complex environments, to achieve the accuracy of camera extrinsic parameter calculation in harsh environments, to ensure the success rate and accuracy of video splicing, and to effectively avoid video splicing failure or obvious splicing marks in complex environments.

[0164] In this embodiment, a video frame to be stitched is acquired from a camera; image feature points are identified from the video frame; spatial feature points corresponding to the image feature points are matched; feature point pairs composed of image feature points and spatial feature points are filtered to obtain target feature point pairs; the camera's extrinsic parameters are calculated based on the target feature point pairs; and the video frame to be stitched is stitched together based on the camera's extrinsic parameters to obtain the stitched target video frame. This embodiment improves the accuracy of camera extrinsic parameter calculation in harsh environments by identifying image feature points in the video frame to be stitched, matching corresponding spatial feature points, and filtering feature point pairs composed of image feature points and spatial feature points. This removes coarse noise points generated during feature point extraction and matching, ensuring the success rate and accuracy of subsequent video frame stitching and effectively avoiding stitching failures or obvious stitching artifacts.

[0165] Based on the above embodiments, in order to enable those skilled in the art to have a deeper understanding of an embodiment of a video splicing method of this application, the complete process of an embodiment of a video splicing method will be described below.

[0166] Reference Figure 5 This is a detailed flowchart of an embodiment of a video splicing method according to this application.

[0167] like Figure 5 As shown, the video frame to be stitched from the current frame acquired by camera 1 is CH1, the video frame to be stitched from the current frame acquired by camera 2 is CH2, and so on, and several cameras can be set up. The following explanation focuses on the processing of CH1; the processing of the other video frames to be stitched is similar and will not be repeated here.

[0168] First, obtain the spatial feature points of the real objects corresponding to the image features. After distortion correction and other processing, extract the coordinates of the image feature points of the identification cards in the image of CH1. Then, match the actual coordinates of the identification cards in the world coordinate system, i.e., the spatial feature points, to obtain 2D-3D feature point pairs of image and spatial feature points. Use the RANSAC method to iteratively filter the feature point pairs to obtain target feature point pairs. If there are more than 4 target feature point pairs, use the PNP method to calculate the extrinsic parameters of camera 1 based on the target feature point pairs. For the calculated extrinsic parameters, continue to calculate the reprojection error and determine whether it meets the preset error threshold. If it does not meet the threshold, optimize the distribution of the extrinsic parameters; if it does, further process the video frames to be stitched based on the extrinsic parameters. Perform an affine transformation on the video frames to be stitched from camera 1 based on the rotation and translation matrices in the extrinsic parameters of camera 1; perform an affine transformation on the video frames to be stitched from camera 2 based on the rotation and translation matrices in the extrinsic parameters of camera 2, and so on, to obtain several transformed video frames. Based on the translation matrices corresponding to several cameras, the video images from multiple cameras are registered and fused to complete the stitching of multiple video images and obtain the stitched target video image.

[0169] In this embodiment, a video frame to be stitched is acquired from a camera; image feature points are identified from the video frame; spatial feature points corresponding to the image feature points are matched; feature point pairs composed of image feature points and spatial feature points are filtered to obtain target feature point pairs; the camera's extrinsic parameters are calculated based on the target feature point pairs; and the video frame to be stitched is stitched together based on the camera's extrinsic parameters to obtain the stitched target video frame. This embodiment improves the accuracy of camera extrinsic parameter calculation in harsh environments by identifying image feature points in the video frame to be stitched, matching corresponding spatial feature points, and filtering feature point pairs composed of image feature points and spatial feature points. This removes coarse noise points generated during feature point extraction and matching, ensuring the success rate and accuracy of subsequent video frame stitching and effectively avoiding stitching failures or obvious stitching artifacts.

[0170] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0171] Based on the above embodiments, this embodiment also provides a video splicing device, which can be applied to terminal devices, servers and other electronic devices.

[0172] Reference Figure 6 The diagram illustrates a structural block diagram of an embodiment of a video splicing device according to this application, which may specifically include the following modules:

[0173] The video frame acquisition module 601 is used to acquire the video frames to be stitched captured by the camera;

[0174] The image feature point recognition module 602 is used to identify image feature points from the video frames to be stitched together.

[0175] The spatial feature point matching module 603 is used to match the spatial feature points corresponding to the image feature points based on the image feature points;

[0176] The feature point pair filtering module 604 is used to filter feature point pairs composed of the image feature points and the spatial feature points to obtain target feature point pairs.

[0177] The extrinsic parameter calculation module 605 is used to calculate the extrinsic parameters of the camera based on the target feature point pair;

[0178] The video splicing module 606 is used to splice the video frames to be spliced ​​according to the external parameters of the camera to obtain the spliced ​​target video frames.

[0179] In one optional embodiment of this application, the image feature point recognition module 602 includes:

[0180] The distortion correction submodule is used to perform distortion correction processing on the video frames to be stitched together to obtain the distortion-corrected video frames.

[0181] The recognition model processing submodule is used to identify the image features in the distortion-free video frame using a recognition model; the recognition model is a model trained using training images containing the image features.

[0182] The image feature point extraction submodule is used to obtain the image feature points based on the image features. In an optional embodiment of this application, the image feature point extraction submodule is further used to...

[0183] Based on the image features, the distortion-free video image is cropped to obtain a feature thumbnail that includes the image features;

[0184] Extract the image feature points from the feature thumbnail;

[0185] The spatial feature point matching module 603 includes:

[0186] The spatial feature point acquisition submodule is used to acquire the spatial feature points corresponding to the image features;

[0187] The reference image acquisition submodule is used to acquire a reference image containing reference feature points; there is a first pairing relationship between the coordinate system of the reference image and the spatial coordinate system, and the spatial coordinate system is the coordinate system where the spatial feature points are located;

[0188] The feature corner point matching submodule is used to match the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point;

[0189] The spatial feature point pairing submodule is used to match the spatial feature points according to the first pairing relationship and the second pairing relationship.

[0190] In an optional embodiment of this application, the feature point pair filtering module 604 includes:

[0191] The iteration count setting submodule is used to set the number of iterations;

[0192] The feature point pair filtering submodule is used to filter out noise point pairs in the feature point pairs according to the number of iterations and using a random sample consensus algorithm to obtain the target feature point pairs.

[0193] In an optional embodiment of this application, the apparatus further includes:

[0194] The feature point pair classification module is used to classify the target feature point pairs to obtain a first feature point pair, a second feature point pair, and a third feature point pair;

[0195] The first projection error calculation module is used to calculate the first projection error based on the first feature point pair, the second feature point pair and the third feature point pair.

[0196] The second projection error calculation module is used to calculate the second projection error based on the first feature point pair and the second feature point pair if the first projection error is greater than a preset error threshold.

[0197] The third projection error calculation module is used to calculate the third projection error based on the first feature point pair if the second projection error is greater than the preset error threshold.

[0198] The extrinsic parameter optimization module is used to correct the extrinsic parameters of the camera to those calculated based on the previous video frame if the third projection error is greater than the preset error threshold.

[0199] In one optional embodiment of this application, the extrinsic parameters of the camera include a rotation matrix and a translation matrix, and the video stitching module 606 includes:

[0200] The affine transformation submodule is used to perform an affine transformation on the distortion-free video frame using the rotation matrix and the translation matrix to obtain the transformed video frame.

[0201] The overlapping region calculation submodule is used to obtain the overlapping region of the transformed video frame based on the translation matrix.

[0202] The video stitching submodule is used to stitch the transformed video frames according to the overlapping areas to obtain the stitched target video frames.

[0203] In one optional embodiment of this application, applied to a coal mine monitoring system, the video footage to be stitched includes a coal mining machine, the coal mining machine includes a body structure and an identification card on the body, the identification card having a unique number, and the image feature point recognition module 602 includes:

[0204] The feature recognition submodule is used to identify the fuselage structure and the number of the identification card from the video frame to be stitched together;

[0205] The feature coordinate acquisition submodule is used to acquire the position coordinates of the fuselage structure and the identification card in the video frame to be stitched together;

[0206] The feature point determination submodule is used to identify the image feature points from the video frames to be stitched together based on the position coordinates.

[0207] In this embodiment, a video frame to be stitched is acquired from a camera; image feature points are identified from the video frame; spatial feature points corresponding to the image feature points are matched; feature point pairs composed of image feature points and spatial feature points are filtered to obtain target feature point pairs; the camera's extrinsic parameters are calculated based on the target feature point pairs; and the video frame to be stitched is stitched together based on the camera's extrinsic parameters to obtain the stitched target video frame. This embodiment improves the accuracy of camera extrinsic parameter calculation in harsh environments by identifying image feature points in the video frame to be stitched, matching corresponding spatial feature points, and filtering feature point pairs composed of image feature points and spatial feature points. This removes coarse noise points generated during feature point extraction and matching, ensuring the success rate and accuracy of subsequent video frame stitching and effectively avoiding stitching failures or obvious stitching artifacts.

[0208] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0209] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0210] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 7 An exemplary apparatus 700 is schematically shown that can be used to implement the various embodiments described in this application.

[0211] In one embodiment, Figure 7 An exemplary device 700 is shown, which includes one or more processors 702, a control module (chipset) 704 coupled to at least one of the processors 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0212] Processor 702 may include one or more single-core or multi-core processors, and processor 702 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 700 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0213] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage device 708) having instructions 714 and one or more processors 702 that are combined with the one or more computer-readable media and configured to execute the instructions 714 to implement the module and thus perform the actions described in this disclosure.

[0214] In one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processors 702 and / or any suitable device or component communicating with the control module 704.

[0215] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0216] Memory 706 may be used, for example, to load and store data and / or instructions 714 for device 700. In one embodiment, memory 706 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 706 may include double data rate type 4 synchronous dynamic random access memory (DDR4 SDRAM).

[0217] In one embodiment, the control module 704 may include one or more input / output controllers to provide an interface to the NVM / storage device 708 and (one or more) input / output devices 710.

[0218] For example, NVM / storage device 708 may be used to store data and / or instructions 714. NVM / storage device 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0219] NVM / storage device 708 may include storage resources that are physically part of a device on which device 700 is mounted, or that can be accessed by the device without needing to be part of the device. For example, NVM / storage device 708 may be accessed via a network via one or more input / output devices 710.

[0220] One or more input / output devices 710 may provide an interface for device 700 to communicate with any other suitable device. Input / output devices 710 may include communication components, audio components, sensor components, etc. A network interface 712 may provide an interface for device 700 to communicate via one or more networks. Device 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0221] In one embodiment, at least one of the processors 702 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 704. In one embodiment, at least one of the processors 702 may be logically packaged with one or more controllers of the control module 704 to form a system-in-package (SiP). In one embodiment, at least one of the processors 702 may be integrated with the logic of one or more controllers of the control module 704 on the same die. In one embodiment, at least one of the processors 702 may be integrated with the logic of one or more controllers of the control module 704 on the same die to form a system-on-a-chip (SoC).

[0222] In various embodiments, device 700 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 700 may have more or fewer components and / or different architectures. For example, in some embodiments, device 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0223] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0224] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0225] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0226] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable video splicing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable video splicing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0227] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable video splicing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0228] These computer program instructions can also be loaded onto a computer or other programmable video splicing terminal device, causing a series of operational steps to be executed on the computer or other programmable terminal device to produce computer-implemented processing, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0229] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0230] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0231] The present application provides a detailed description of a video splicing method and apparatus, an electronic device, and a storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. A method for stitching video frames, characterized in that, The method includes: Acquire video footage captured by the camera to be stitched together; Identify image feature points from the video frames to be stitched together; Based on the image feature points, match the corresponding spatial feature points, where the spatial feature points are obtained in advance by measuring the spatial coordinates of the real objects corresponding to the image features; By filtering the feature point pairs formed by the image feature points and the spatial feature points, the target feature point pairs are obtained; Calculate the extrinsic parameters of the camera based on the target feature point pairs; Based on the external parameters of the camera, the video frames to be stitched are spliced ​​together to obtain the spliced ​​target video frame; The step of identifying image feature points from the video frames to be stitched includes: The video frames to be stitched are subjected to distortion correction processing to obtain the distortion-corrected video frames; A recognition model is used to identify the image features in the distortion-free video frame; the recognition model is a model trained using training images containing the image features. Based on the image features, the image feature points are obtained; The step of obtaining the image feature points based on the image features includes: Based on the image features, the distortion-free video image is cropped to obtain a feature thumbnail that includes the image features; Extract the feature corner points of the feature thumbnail as the image feature points; The step of matching spatial feature points corresponding to the image feature points includes: Obtain the spatial feature points corresponding to the image features; Obtain a reference image containing reference feature points; the coordinate system of the reference image and the spatial coordinate system have a first pairing relationship, and the spatial coordinate system is the coordinate system where the spatial feature points are located; Match the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point; The spatial feature points are matched based on the first pairing relationship and the second pairing relationship.

2. The method according to claim 1, characterized in that, The step of filtering the feature point pairs composed of the image feature points and the spatial feature points to obtain the target feature point pairs includes: Set the number of iterations; Based on the number of iterations, a random sample consensus algorithm is used to filter out noise point pairs in the feature point pairs to obtain the target feature point pairs.

3. The method according to claim 1, characterized in that, After calculating the extrinsic parameters of the camera based on the target feature point pairs, the method further includes: The target feature point pairs are classified to obtain a first feature point pair, a second feature point pair, and a third feature point pair; The first reprojection error is calculated based on the first feature point pair, the second feature point pair, and the third feature point pair. If the first projection error is greater than a preset error threshold, then the second projection error is calculated based on the first feature point pair and the second feature point pair. If the second projection error is greater than the preset error threshold, then the third projection error is calculated based on the first feature point pair. If the third projection error is greater than the preset error threshold, the extrinsic parameters of the camera are corrected to the extrinsic parameters calculated based on the previous video frame.

4. The method according to claim 1, characterized in that, The camera's extrinsic parameters include a rotation matrix and a translation matrix. The step of stitching together the video frames to be stitched based on the camera's extrinsic parameters to obtain the stitched target video frame includes: Using the rotation matrix and the translation matrix, an affine transformation is performed on the distortion-free video frame to obtain the transformed video frame; Based on the translation matrix, the overlapping area of ​​the transformed video frame is obtained; Based on the overlapping area, the transformed video frames are stitched together to obtain the stitched target video frame.

5. The method according to claim 1, characterized in that, Applied to a coal mine monitoring system, the video footage to be stitched together includes a coal mining machine, which comprises a machine body structure and an identification card on the machine body. The identification card has a unique number. Identifying image feature points from the video footage to be stitched together includes: Identify the fuselage structure and the number on the identification card from the video footage to be stitched together; Obtain the position coordinates of the fuselage structure and the identification card in the video frame to be stitched together; Based on the location coordinates, the image feature points are identified from the video frames to be stitched together.

6. A video splicing device, characterized in that, The device includes: The video capture module is used to acquire video footage captured by the camera for stitching. An image feature point recognition module is used to identify image feature points from the video frames to be stitched together. The spatial feature point calculation module is used to match the spatial feature points corresponding to the image feature points based on the image feature points. The spatial feature points are obtained in advance by measuring the spatial coordinates of the real objects corresponding to the image features. The feature point pair filtering module is used to filter feature point pairs composed of the image feature points and the spatial feature points to obtain target feature point pairs. The extrinsic parameter calculation module is used to calculate the extrinsic parameters of the camera based on the target feature point pairs; The video splicing module is used to splice the video frames to be spliced ​​according to the external parameters of the camera to obtain the spliced ​​target video frames; The image feature point recognition module includes: The distortion correction submodule is used to perform distortion correction processing on the video frames to be stitched together to obtain the distortion-corrected video frames. The recognition model processing submodule is used to identify the image features in the distortion-free video frame using a recognition model; the recognition model is a model trained using training images containing the image features. The image feature point extraction submodule is used to obtain the image feature points based on the image features; The image feature point extraction submodule is specifically used for: Based on the image features, the distortion-free video image is cropped to obtain a feature thumbnail that includes the image features; Extract the feature corner points of the feature thumbnail as the image feature points; The spatial feature point matching module includes: The spatial feature point acquisition submodule is used to acquire the spatial feature points corresponding to the image features; The reference image acquisition submodule is used to acquire a reference image containing reference feature points; there is a first pairing relationship between the coordinate system of the reference image and the spatial coordinate system, and the spatial coordinate system is the coordinate system where the spatial feature points are located; The feature corner point matching submodule is used to match the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point; The spatial feature point pairing submodule is used to match the spatial feature points according to the first pairing relationship and the second pairing relationship.

7. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the video stitching method as described in one or more of claims 1-5.

8. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the video stitching method as described in one or more of claims 1-5.

Images