Video picture splicing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a video picture splicing method and device, equipment and a storage medium. The method comprises the following steps: acquiring a to-be-spliced video picture shot by a camera; identifying image feature points from the video image to be spliced; matching spatial feature points corresponding to the image feature points according to the image feature points; screening feature point pairs formed by the image feature points and the spatial feature points to obtain target feature point pairs; calculating external parameters of the camera according to the target feature point pair; and splicing the to-be-spliced video picture according to the external parameters of the camera to obtain a spliced target video picture. According to the embodiment of the invention, image feature point identification is carried out on the to-be-spliced video image, the corresponding spatial feature points are matched, the feature point pairs composed of the image feature points and the spatial feature points are screened, and coarse noise points generated when the matched feature points are extracted are filtered out, so that the camera external parameter calculation accuracy in a severe environment is improved; and the success rate and the precision of follow-up video picture splicing are ensured.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video picture splicing method and device, an electronic device and a storage medium. Background Art

[0002] Realizing a production environment without human supervision in dangerous environments such as underground mines is an important development direction in the mining industry. By introducing advanced automation technology and artificial intelligence, the working status of coal miners can be monitored in real time, significantly reducing the risk of personal injury faced by miners in the mines, thereby improving working conditions and saving labor resources.

[0003] In recent years, monocular cameras have attracted more and more attention due to their easy installation and low cost. Monocular camera video stitching technology uses a multi-camera system to capture device images from different perspectives and automatically stitches them into one or more wide-field or panoramic images through specific image algorithms. This method not only significantly reduces equipment costs, but also is suitable for more flexible application scenarios.

[0004] Although the video stitching technology based on monocular cameras has made certain progress in related fields, it still faces challenges in real-time performance, stitching accuracy, and coping with complex environmental changes. In the application of coal mine monitoring, the coal dust generated during the operation of the coal mining machine will interfere with the camera shooting picture, resulting in more bad points when extracting the feature points of the coal mining machine body for matching algorithm, which in turn affects the calculation accuracy of the camera external parameters, and may eventually lead to stitching failure or obvious stitching marks. Summary of the invention

[0005] The embodiment of the present application provides a video picture stitching method to solve or at least partially solve the above-mentioned problem.

[0006] Correspondingly, the embodiment of the present application also provides a video screen splicing device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the present application discloses a video image stitching method, which includes:

[0008] Get the video images to be stitched taken by the camera;

[0009] Identifying image feature points from the video images to be spliced;

[0010] According to the image feature points, matching the spatial feature points corresponding to the image feature points;

[0011] Filtering feature point pairs consisting of the image feature points and the space feature points to obtain target feature point pairs;

[0012] Calculating the extrinsic parameters of the camera according to the target feature point pair;

[0013] The video images to be spliced ​​are spliced ​​according to the external parameters of the camera to obtain a spliced ​​target video image.

[0014] In an optional embodiment of the present application, the identifying image feature points from the video pictures to be spliced ​​includes:

[0015] Performing a dedistortion process on the video images to be spliced ​​to obtain dedistorted video images;

[0016] Using a recognition model to recognize picture features in the dedistorted video picture; the recognition model is a model trained using a training image containing the picture features;

[0017] The image feature points are obtained according to the picture features.

[0018] In an optional embodiment of the present application, obtaining the image feature points according to the picture features includes:

[0019] According to the picture features, cropping the dedistorted video picture to obtain a feature thumbnail including the picture features;

[0020] Extracting the image feature points of the feature thumbnail;

[0021] The matching of the spatial feature points corresponding to the image feature points according to the image feature points includes:

[0022] Acquire the spatial feature point corresponding to the picture feature;

[0023] Acquire a reference image containing reference feature points; there is a first pairing relationship between the coordinate system of the reference image and the spatial coordinate system, and the spatial coordinate system is the coordinate system where the spatial feature points are located;

[0024] Matching the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point;

[0025] The spatial feature points are matched according to the first pairing relationship and the second pairing relationship.

[0026] In an optional embodiment of the present application, the screening of the feature point pairs consisting of the image feature points and the space feature points to obtain the target feature point pairs includes:

[0027] Set the number of iterations;

[0028] According to the number of iterations, a random sample consensus algorithm is used to filter noise point pairs in the feature point pairs to obtain target feature point pairs.

[0029] In an optional embodiment of the present application, after calculating the external parameters of the camera according to the target feature point pair, the method further includes:

[0030] Classifying the target feature point pairs to obtain a first feature point pair, a second feature point pair, and a third feature point pair;

[0031] Calculate a first reprojection error based on the first feature point pair, the second feature point pair, and the third feature point pair;

[0032] If the first reprojection error is greater than a preset error threshold, calculating a second reprojection error according to the first feature point pair and the second feature point pair;

[0033] If the second reprojection error is greater than the preset error threshold, calculating a third reprojection error based on the first feature point pair;

[0034] If the third reprojection error is greater than the preset error threshold, the extrinsic parameter of the camera is corrected to the extrinsic parameter calculated according to the previous frame of video.

[0035] In an optional embodiment of the present application, the external parameters of the camera include a rotation matrix and a translation matrix, and the video images to be spliced ​​are spliced ​​according to the external parameters of the camera to obtain the spliced ​​target video images, including:

[0036] Using the rotation matrix and the translation matrix, performing an affine transformation on the dedistorted video picture to obtain a transformed video picture;

[0037] Obtaining an overlapping area of ​​the transformed video picture according to the translation matrix;

[0038] The transformed video images are spliced ​​according to the overlapping areas to obtain the spliced ​​target video images.

[0039] In an optional embodiment of the present application, the method is applied to a coal mine monitoring system, the video screen to be spliced ​​includes a coal mining machine, the coal mining machine includes a fuselage structure and an identification card on the fuselage, the identification card has a unique number, and the identifying image feature points from the video screen to be spliced ​​includes:

[0040] Identify the fuselage structure and the number of the identification card from the video picture to be spliced;

[0041] Obtaining the position coordinates of the fuselage structure and the identification card in the video picture to be spliced ​​respectively;

[0042] According to the position coordinates, the image feature points are identified from the video images to be spliced.

[0043] The embodiment of the present application also discloses a video picture splicing device, the device comprising:

[0044] The video image acquisition module is used to acquire the video images to be spliced ​​taken by the camera;

[0045] An image feature point recognition module, used to recognize image feature points from the video images to be spliced;

[0046] A spatial feature point calculation module, used to match the spatial feature points corresponding to the image feature points according to the image feature points;

[0047] A feature point pair screening module, used for screening the feature point pairs consisting of the image feature points and the space feature points to obtain target feature point pairs;

[0048] An extrinsic parameter calculation module, used to calculate the extrinsic parameters of the camera according to the target feature point pair;

[0049] The video picture splicing module is used to splice the video pictures to be spliced ​​according to the external parameters of the camera to obtain the spliced ​​target video pictures.

[0050] The embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the video screen stitching method as described in one or more of the embodiments of the present application.

[0051] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, a processor executes the video picture splicing method as described in one or more of the embodiments of the present application.

[0052] Compared with the prior art, the embodiments of the present application have the following advantages:

[0053] In an embodiment of the present application, a video screen to be spliced ​​taken by a camera is obtained; image feature points are identified from the video screen to be spliced; according to the image feature points, spatial feature points corresponding to the image feature points are matched; feature point pairs consisting of image feature points and spatial feature points are screened to obtain target feature point pairs; according to the target feature point pairs, the external parameters of the camera are calculated; according to the external parameters of the camera, the video screen to be spliced ​​is spliced ​​to obtain the target video screen after splicing. The embodiment of the present application performs image feature point identification on the video screen to be spliced, matches the corresponding spatial feature points, screens the feature point pairs consisting of image feature points and spatial feature points, and filters out the coarse noise points generated when extracting the matching feature points, thereby improving the accuracy of the camera external parameter calculation in harsh environments, ensuring the success rate and accuracy of subsequent video screen splicing, and effectively avoiding splicing failures or obvious splicing marks. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of a video image splicing method embodiment of the present application;

[0055] Figure 2 This is a feature point matching flow chart of an embodiment of a video picture splicing method of the present application;

[0056] Figure 3 This is a flow chart of obtaining feature point pairs in an embodiment of a video picture stitching method of the present application;

[0057] Figure 4 This is an external parameter optimization flow chart of an embodiment of a video screen splicing method of the present application;

[0058] Figure 5 This is a specific flow chart of an embodiment of a video screen splicing method of the present application;

[0059] Figure 6 It is a structural block diagram of an embodiment of a video picture splicing device of the present application;

[0060] Figure 7 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0062] Reference Figure 1 , is a flowchart of a video screen splicing method embodiment of the present application, comprising the following steps:

[0063] Step 101: Obtain video images to be stitched that are shot by a camera.

[0064] In an embodiment of the present application, before step 101, multiple monocular cameras can be placed at fixed intervals in the area where monitoring and shooting is required, and the distances from all cameras to the devices to be detected are ensured to be equal, and the straight-line distances from the cameras to the devices are recorded.

[0065] In one embodiment, the area to be monitored and photographed may be a coal mine, and the equipment to be detected may be a coal mining machine, wherein the coal mining machine may be divided into three parts, namely, a primary main body, a secondary body that can swing up and down, and a drum cutting part.

[0066] In step 101, video images to be spliced ​​of current frames shot by multiple cameras are obtained.

[0067] Step 102: Identify image feature points from the video images to be spliced.

[0068] In an optional embodiment, an identification card can be pasted on the surface of the device to be tested in advance, and the spatial coordinates of the four corner points of the identification card and each body structure of the device in the spatial coordinate system can be recorded as spatial feature points. The posted identification card includes a directly identifiable number and the shape combination of the identification card is unique.

[0069] Furthermore, image data of the identification card and each body structure of the device taken from multiple angles can be collected, and after the image coordinate data is annotated, a suitable neural network is selected, and the neural network is trained according to the image data to obtain an image recognition model. The image recognition model can recognize the identification card and each body structure of the device from the video screen.

[0070] In step 102, an image recognition model may be used to identify relevant image features from the video images to be spliced, and the video images to be spliced ​​may be cropped according to the identified image features to obtain a feature thumbnail, and then feature corner points in the feature thumbnail may be extracted as image feature points of the frame. In one embodiment, the image features may include the coordinate position and corresponding number of the identification card, and various body structures of the device.

[0071] Step 103: Calculate the spatial feature points corresponding to the image feature points according to the image feature points.

[0072] In step 103, the spatial feature points corresponding to the image feature points are matched according to the image feature points to obtain 2D-3D feature point pairs, wherein the spatial feature points are obtained by pre-measuring the spatial coordinates of the physical objects corresponding to the image features.

[0073] Step 104: Filter the feature point pairs consisting of the image feature points and the space feature points to obtain target feature point pairs.

[0074] In step 104, a target feature point pair is selected from the feature point pairs consisting of the image feature points and the spatial feature points. Specifically, the number of iterations can be set to perform RANSAC (Random Sample Consensus) sampling calculation to select the internal points and filter out the external points with large noise, thereby obtaining the target feature point pair.

[0075] In one embodiment, if the obtained target feature point pairs are less than 4 pairs, the extrinsic parameters of the frame are not updated, that is, the target feature point pairs of the frame are not used to calculate the extrinsic parameters of the frame, but the extrinsic parameters calculated by the previous frame of the camera are used.

[0076] Step 105: Calculate the extrinsic parameters of the camera according to the target feature point pair.

[0077] In step 105, the external parameters of the camera are calculated according to the target feature point pairs, that is, the mapping relationship between the coordinates of the feature points in the world coordinate system and the coordinates of the image coordinate system is established.

[0078] Specifically, in an optional embodiment, the camera extrinsic parameters can be calculated using the PNP method (Perspective-n-Point, a method for calculating the camera pose from multiple points in a three-dimensional space). From the perspective projection model, according to formula (1), the pixel coordinate system of the image ( uv uv) and the camera coordinate system (O c X c Y c Z c )

[0079]

[0080] Among them, u and v are the image coordinates of the image feature points; K is the intrinsic parameter matrix of the camera; X c , Y c , Z c is the coordinate of the camera coordinate system.

[0081] Therefore, according to formula (2), the three-dimensional coordinates of the imaging point on the normalized image plane can be obtained:

[0082]

[0083] Among them, X1, Y1, and Z1 are the three-dimensional coordinates of the imaging point on the normalized image plane; u and v are the image coordinates of the image feature points; and K is the intrinsic parameter matrix of the camera.

[0084] According to the perspective projection model, the pixel coordinate system (O uv uv) and the normalized image plane coordinate system, and further obtain the camera coordinate system (Oc X c Y c Z c ) and the world coordinate system (O w X w Y w Z w )

[0085]

[0086] Among them, R and t are the external parameters of the camera, R is the rotation matrix, and T is the translation matrix; X c , Y c , Z c is the coordinate of the camera coordinate system; X w , Y w , Z w is the coordinate of the world coordinate system, that is, the coordinate of the spatial feature point.

[0087] Therefore, when the camera intrinsic parameters are known, the extrinsic rotation matrix and translation matrix of each monocular camera in the current frame can be calculated through the target feature point pairs.

[0088] Step 106: splice the video images to be spliced ​​according to the external parameters of the camera to obtain a spliced ​​target video image.

[0089] In step 106, the video images to be spliced ​​of the current frames acquired by several cameras are spliced ​​according to the calculated external parameters of the cameras to obtain a spliced ​​target video image.

[0090] In an embodiment of the present application, a video screen to be spliced ​​taken by a camera is obtained; image feature points are identified from the video screen to be spliced; according to the image feature points, spatial feature points corresponding to the image feature points are matched; feature point pairs consisting of image feature points and spatial feature points are screened to obtain target feature point pairs; according to the target feature point pairs, the external parameters of the camera are calculated; according to the external parameters of the camera, the video screen to be spliced ​​is spliced ​​to obtain the target video screen after splicing. The embodiment of the present application performs image feature point identification on the video screen to be spliced, matches the corresponding spatial feature points, screens the feature point pairs consisting of image feature points and spatial feature points, and filters out the coarse noise points generated when extracting the matching feature points, thereby improving the accuracy of the camera external parameter calculation in harsh environments, ensuring the success rate and accuracy of subsequent video screen splicing, and effectively avoiding splicing failures or obvious splicing marks.

[0091] In an optional embodiment of the present application, step 102 includes:

[0092] Performing a dedistortion process on the video images to be spliced ​​to obtain dedistorted video images;

[0093] Using a recognition model to recognize picture features in the dedistorted video picture; the recognition model is a model trained using a training image containing the picture features;

[0094] The image feature points are obtained according to the picture features.

[0095] In this embodiment, before identifying the image features in the video images, it is necessary to first perform a dedistortion process on the video images to be spliced ​​to obtain the dedistorted video images, and then use a recognition model to identify the image features in the dedistorted video images, wherein the recognition model is a model trained using training images containing image features. In one embodiment, the image features may include the position coordinates and corresponding numbers of the identification card of a coal mining machine in a coal mine, and the various fuselage structures of the equipment.

[0096] Furthermore, image feature points may be extracted from the video picture according to the identified picture features in the dedistorted video picture.

[0097] In one embodiment, image data of identification cards and various body structures of the device taken from multiple angles may be collected, and after the image coordinate data is annotated, a suitable neural network is selected, and the neural network is trained according to the image data to obtain an image recognition model.

[0098] In another embodiment, if the equipment to be inspected is a coal mining machine, since the drum cutting part generally cannot be photographed when the coal mining machine is working, the primary main body and the secondary body that can swing up and down of the coal mining machine can be used as the equipment body structure in the image data for training.

[0099] In another embodiment, in order to ensure the real-time splicing of the video images, the image features in the dedistorted video images may be identified by using a YOLO (You Only Look Once, a target detection algorithm) detection method with fast detection speed and strong real-time performance.

[0100] The embodiment of the present application can use a recognition model to identify the image features in the dedistorted video image, and then obtain image feature points, which can improve the recognition accuracy of the image feature points, and thus be used for the subsequent calculation of camera extrinsic parameters.

[0101] In an optional embodiment of the present application, obtaining the image feature points according to the picture features includes:

[0102] According to the picture features, cropping the dedistorted video picture to obtain a feature thumbnail including the picture features;

[0103] Extracting the image feature points of the feature thumbnail;

[0104] The step 103 includes:

[0105] Acquire the spatial feature point corresponding to the picture feature;

[0106] Acquire a reference image containing reference feature points; there is a first pairing relationship between the coordinate system of the reference image and the spatial coordinate system, and the spatial coordinate system is the coordinate system where the spatial feature points are located;

[0107] Matching the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point;

[0108] The spatial feature points are matched according to the first pairing relationship and the second pairing relationship.

[0109] In this embodiment, image feature points are extracted from the dedistorted video image, and corresponding spatial feature points are matched according to the image feature points. Specifically, the following steps may be included.

[0110] When the embodiment of the present application is applied to a coal mine monitoring system, due to the low visibility in the mine, the presence of a large number of similar structures in the picture and the large interference of coal ash and dust, the method of extracting feature corner points from the entire picture and matching them with reference feature points will introduce serious mismatching problems, which will have a great impact on the subsequent calculation of external parameters of the camera and video picture stitching. Therefore, the distorted video picture can be cropped according to the picture features to obtain a feature sub-image including the picture features, and then the feature corner points of the feature sub-image can be extracted as image feature points.

[0111] Obtain the spatial feature points corresponding to the physical object of the previously measured picture features, and the reference image containing the reference feature points. There is a first pairing relationship between the coordinate system of the reference image and the spatial coordinate system, and the spatial coordinate system is the coordinate system where the spatial feature points are located. Specifically, in an optional embodiment, in advance, according to formula (4), the size conversion relationship between the pixel point and the actual length in the space can be calculated, and the size conversion relationship can be obtained as follows:

[0112]

[0113] Among them, FOV is the viewing angle of the monocular camera; L is the maximum length of the world coordinate system that the camera can capture; d is the distance from the camera to the device.

[0114] Then, the size conversion between the world coordinate system and the image plane can be obtained according to formula (5), that is, a pixel point in the image corresponds to the length (cm, centimeters) in the world coordinate system:

[0115]

[0116] The resolution of the monocular camera image is W×H pixels; FOV is the viewing angle of the monocular camera; and d is the distance from the camera to the device.

[0117] Therefore, based on the relationship between a pixel point in the image and the length in the world coordinate system, the coordinate information of the reference feature point on the reference image corresponding to the actual device feature point in the world coordinate system can be obtained, that is, the first pairing relationship between the coordinate system of the reference image and the space coordinate system.

[0118] The image feature points of the feature thumbnail are matched with the reference feature points by a brute force matching method to obtain a second pairing relationship between the image feature points and the reference feature points, i.e., a 2D-2D feature point pair. Thus, the spatial feature points corresponding to the image feature points, i.e., a 2D-3D feature point pair, can be matched according to the first pairing relationship and the second pairing relationship.

[0119] Reference Figure 2 , is a feature point matching flow chart of an embodiment of a video picture stitching method of the present application.

[0120] like Figure 2 As shown, when an identification card containing a number is pasted on the surface of the device to be detected, the image recognition model can be pre-trained using training images of a group of identification cards at different angles, and the label of the training image is the number of the identification card, thereby obtaining a trained image recognition model.

[0121] Get the current frame of the single-channel camera after de-distortion processing. Use the trained image recognition model to detect the identification card number in the picture and the corresponding position coordinates of the identification card as picture features. According to the position coordinates corresponding to the detected identification card, expand it by 1.1 times, crop the feature thumbnail in the video picture, pass the coordinates of the feature corner points extracted from the identification card feature thumbnail as image feature points, and extract the reference feature point coordinates of the reference image. The second pairing relationship between the image feature points and the reference feature points, i.e., 2D-2D feature point pairs, is obtained by brute force matching.

[0122] Figure 2 The specific values ​​for feature thumbnail cropping are only for example, and those skilled in the art can set corresponding values ​​according to actual needs.

[0123] Reference Figure 3 , is a feature point pair acquisition flow chart of an embodiment of a video picture stitching method of the present application.

[0124] like Figure 3As shown, the second pairing relationship between the feature corner points of the feature sub-image cropped from the camera video screen and the reference feature points in the reference image, i.e., the 2D-2D feature point pair, and the relationship between a pixel point in the image and the length in the world coordinate system, the reference feature points on the reference image corresponding to the actual device feature points in the world coordinate system, the real coordinate information of the feature points of the identification card, have a first pairing relationship, i.e., the 2D-3D correspondence relationship. Then, according to the first pairing relationship and the second pairing relationship, the spatial feature points corresponding to the image feature points can be matched, i.e., the 2D-3D feature point pairs corresponding to the image coordinate system-space coordinate system.

[0125] The embodiment of the present application crops the feature small image and then extracts the image feature points. There is no need to extract image feature points for the entire video screen, which reduces resource consumption when extracting image feature points and improves the speed of image feature point extraction. The feature corner points of the feature small image are matched with the reference feature points to avoid the mismatch problem that may be caused by directly matching the feature corner points of the entire image with the reference feature points, thereby obtaining a more accurate feature point pair for subsequent calculation of the camera's external parameters.

[0126] In an optional embodiment of the present application, step 104 includes:

[0127] Set the number of iterations;

[0128] According to the number of iterations, a random sample consensus algorithm is used to filter noise point pairs in the feature point pairs to obtain target feature point pairs.

[0129] In this embodiment, a random sample consensus algorithm can be calculated, the number of iterations can be set, and the noise point pairs in the feature point pairs can be filtered to obtain the target feature point pairs. Specifically, it can be determined based on formula (6) whether the feature point is an internal point, that is, a feature point in the data set that meets the fitting model, thereby filtering out the target feature point pairs.

[0130]

[0131] Among them, Inlier represents an interior point; Outlier represents an exterior point; t is a threshold value, and t is selected so that the probability that the feature point is an interior point is α, and generally α=95% can be selected, t 2 The value can be determined by reading the table; F and δ are preset parameters; d is the distance from the feature point to the fitting model.

[0132] The embodiment of the present application can further screen the obtained feature point pairs, thereby filtering out the noise point pairs in the feature point pairs and ensuring the accuracy of the camera extrinsic parameter calculation.

[0133] In an optional embodiment of the present application, after step 105, the method further includes:

[0134] Classifying the target feature point pairs to obtain a first feature point pair, a second feature point pair, and a third feature point pair;

[0135] Calculate a first reprojection error based on the first feature point pair, the second feature point pair, and the third feature point pair;

[0136] If the first reprojection error is greater than a preset error threshold, calculating a second reprojection error according to the first feature point pair and the second feature point pair;

[0137] If the second reprojection error is greater than the preset error threshold, calculating a third reprojection error based on the first feature point pair;

[0138] If the third reprojection error is greater than the preset error threshold, the extrinsic parameter of the camera is corrected to the extrinsic parameter calculated according to the previous frame of video.

[0139] In this embodiment, the feature point pairs can be further classified by type according to the objective function of minimizing the reprojection error, and the external parameters can be optimized step by step to improve the accuracy of the external parameter calculation.

[0140] The target feature point pairs are classified to obtain the first feature point pair, the second feature point pair and the third feature point pair. In an optional embodiment, the feature point pair with a clear card surface and a high degree of recognition accuracy can be used as the first feature point pair; the feature point pair with a low degree of recognition accuracy for the primary fuselage but a small degree of change when the device is running can be used as the second feature point pair; the feature point pair with a low degree of recognition accuracy for the secondary fuselage and a change in the up and down swing of the fuselage when the device is running can be used as the third feature point pair.

[0141] Reference Figure 4 , is an external parameter optimization flow chart of an embodiment of a video screen splicing method of the present application.

[0142] In this embodiment, the goal of the reprojection error minimization objective function is to minimize the error between the projection of the spatial feature point on the image and the actual image feature point by optimizing the external parameters of the camera.

[0143] like Figure 4 As shown, first, the first extrinsic parameter of the camera is calculated according to the first feature point pair, the second feature point pair and the third feature point pair, and the first reprojection error is calculated according to the first extrinsic parameter using the PNP method, that is, the projection of the spatial feature point on the image is calculated by the first extrinsic parameter, and the error between the projection of the spatial feature point on the image and the actual image feature point is calculated.

[0144] If the first reprojection error is greater than a preset error threshold, the second extrinsic parameter of the camera is calculated based on the first feature point pair and the second feature point pair, and the second reprojection error is calculated using the PNP method based on the second extrinsic parameter, that is, the projection of the spatial feature point on the image is calculated by the second extrinsic parameter, and the error between the projection of the spatial feature point on the image and the actual image feature point is calculated.

[0145] If the second reprojection error is greater than the preset error threshold, the third extrinsic parameter of the camera is calculated based on the first feature point pair, and the third reprojection error is calculated using the PNP method based on the third extrinsic parameter, that is, the projection of the spatial feature point on the image is calculated by the third extrinsic parameter, and the error between the projection of the spatial feature point on the image and the actual image feature point is calculated.

[0146] If the third reprojection error is still greater than the preset error threshold, the target feature points of this frame are not used to update the camera's external parameters R / T (rotation matrix / translation matrix), but the camera's external parameters are corrected to the external parameters calculated based on the previous frame of video.

[0147] If the first reprojection error is less than the preset error threshold, or the second reprojection error is less than the preset error threshold, or the third reprojection error is less than the preset error threshold, the target feature point of this frame can be used to update the R / T of the camera's extrinsic parameter.

[0148] The embodiment of the present application can optimize the camera's extrinsic parameter calculation in steps according to the reprojection error, thereby ensuring the accuracy of the calculated camera extrinsic parameters, thereby ensuring the success rate and accuracy of subsequent video image stitching, and effectively avoiding stitching failures or obvious stitching marks.

[0149] In an optional embodiment of the present application, the external parameters of the camera include a rotation matrix and a translation matrix, and step 106 includes:

[0150] Using the rotation matrix and the translation matrix, performing an affine transformation on the dedistorted video picture to obtain a transformed video picture;

[0151] Obtaining an overlapping area of ​​the transformed video picture according to the translation matrix;

[0152] The transformed video images are spliced ​​according to the overlapping areas to obtain the spliced ​​target video images.

[0153] In this embodiment, the dedistorted video images may be spliced ​​according to the external parameters of the camera, thereby obtaining a spliced ​​target video image.

[0154] Specifically, the rotation matrix and the translation matrix are used to perform an affine transformation on the dedistorted video image to obtain the transformed video image. In an optional embodiment, since the device is mostly placed in the center of the shooting screen, the transformed video image can be cropped or black-bordered according to actual needs.

[0155] According to the translation matrix, we can know the offset distance of the image feature points of the video screen acquired by each camera relative to the basic origin, and then we can get the overlapping area of ​​the two continuous video screens of the processed channel. In the overlapping area, the weighted average method is used to perform smooth image transition, so as to complete the splicing of the video screens of multiple cameras and obtain the spliced ​​target video screen.

[0156] The embodiment of the present application can splice the dedistorted video images according to the external parameters of the camera, so as to obtain the spliced ​​target video images, thereby ensuring the success rate and accuracy of the video image splicing.

[0157] In an optional embodiment of the present application, which is applied to a coal mine monitoring system, the video image to be spliced ​​includes a coal mining machine, the coal mining machine includes a body structure and an identification card on the body, and the identification card has a unique number, and step 102 includes:

[0158] Identify the fuselage structure and the number of the identification card from the video picture to be spliced;

[0159] Obtaining the position coordinates of the fuselage structure and the identification card in the video picture to be spliced ​​respectively;

[0160] According to the position coordinates, the image feature points are identified from the video images to be spliced.

[0161] In this embodiment, if a video image stitching method shown in this embodiment is applied to a coal mine monitoring system, the area to be monitored and photographed may be a coal mine, and the equipment to be detected may be a coal mining machine. The coal mining machine may be divided into three parts, namely, a primary main body, a secondary body that can swing up and down, and a drum cutting part. An identification card may be affixed to the surface of the coal mining machine in advance, and the posted identification card includes a directly identifiable number and the shape combination of the identification card is unique.

[0162] Therefore, when acquiring image feature points, this embodiment can obtain the picture features of the fuselage structure and the identification card in the video screen to be stitched, respectively, and cut out feature thumbnails according to the picture features, and use the feature corner points in the feature thumbnails and the position coordinates as image feature points to perform subsequent video stitching steps.

[0163] The embodiments of the present application can be applied to coal mining areas with complex environments, achieve the accuracy of camera extrinsic parameter calculation in harsh environments, ensure the success rate and accuracy of video image stitching, and effectively avoid the failure of video frame switching stitching in complex environments or the generation of obvious stitching marks.

[0164] In an embodiment of the present application, a video screen to be spliced ​​taken by a camera is obtained; image feature points are identified from the video screen to be spliced; according to the image feature points, spatial feature points corresponding to the image feature points are matched; feature point pairs consisting of image feature points and spatial feature points are screened to obtain target feature point pairs; according to the target feature point pairs, the external parameters of the camera are calculated; according to the external parameters of the camera, the video screen to be spliced ​​is spliced ​​to obtain the target video screen after splicing. The embodiment of the present application performs image feature point identification on the video screen to be spliced, matches the corresponding spatial feature points, screens the feature point pairs consisting of image feature points and spatial feature points, and filters out the coarse noise points generated when extracting the matching feature points, thereby improving the accuracy of the camera external parameter calculation in harsh environments, ensuring the success rate and accuracy of subsequent video screen splicing, and effectively avoiding splicing failures or obvious splicing marks.

[0165] On the basis of the above embodiments, in order to enable those skilled in the art to have a deeper understanding of an embodiment of a video picture stitching method of the present application, the complete process of an embodiment of a video picture stitching method will be described below.

[0166] Reference Figure 5 , is a specific flow chart of an embodiment of a video screen splicing method of the present application.

[0167] like Figure 5 As shown, the video picture to be spliced ​​of the current frame obtained by camera 1 is CH1, and the video picture to be spliced ​​of the current frame obtained by camera 2 is CH2, and so on. Several cameras can be set. The following is an explanation of the processing of CH1, and the processing methods of the other video pictures to be spliced ​​are similar and will not be repeated here.

[0168] The spatial feature points of the physical object corresponding to the image features are obtained in advance. The image feature point coordinates of the identification card in the image are extracted from CH1 after dedistortion and other processing, and the actual coordinates of the identification card in the world coordinate system, that is, the spatial feature points, are matched according to the image feature points, thereby obtaining 2D-3D feature point pairs of image feature points and spatial feature points. The RANSAC method is used to iteratively filter the feature point pairs to obtain the target feature point pairs. If the target feature point pairs are greater than 4 pairs, the PNP method is used to calculate the external parameters of camera 1 according to the target feature point pairs. For the calculated external parameters, the reprojection error is continued to be calculated to determine whether it meets the preset error threshold. If it does not meet the preset error threshold, the external parameters are distributed optimized. If it meets the preset error threshold, the video images to be spliced ​​are further processed according to the external parameters. According to the rotation matrix and translation matrix in the external parameters of camera 1, the video images to be spliced ​​of camera 1 are affine transformed; according to the rotation matrix and translation matrix in the external parameters of camera 2, the video images to be spliced ​​of camera 2 are affine transformed, and so on, to obtain several transformed video images. According to the translation matrices corresponding to several cameras, the video images of multiple cameras are registered and fused, so as to complete the splicing of multiple video images and obtain the spliced ​​target video image.

[0169] In an embodiment of the present application, a video screen to be spliced ​​taken by a camera is obtained; image feature points are identified from the video screen to be spliced; according to the image feature points, spatial feature points corresponding to the image feature points are matched; feature point pairs consisting of image feature points and spatial feature points are screened to obtain target feature point pairs; according to the target feature point pairs, the external parameters of the camera are calculated; according to the external parameters of the camera, the video screen to be spliced ​​is spliced ​​to obtain the target video screen after splicing. The embodiment of the present application performs image feature point identification on the video screen to be spliced, matches the corresponding spatial feature points, screens the feature point pairs consisting of image feature points and spatial feature points, and filters out the coarse noise points generated when extracting the matching feature points, thereby improving the accuracy of the camera external parameter calculation in harsh environments, ensuring the success rate and accuracy of subsequent video screen splicing, and effectively avoiding splicing failures or obvious splicing marks.

[0170] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0171] On the basis of the above embodiments, this embodiment further provides a video picture splicing device, which is applied to electronic devices such as terminal devices and servers.

[0172] Reference Figure 6 , shows a structural block diagram of an embodiment of a video picture splicing device of the present application, which may specifically include the following modules:

[0173] The video image acquisition module 601 is used to acquire the video images to be spliced ​​taken by the camera;

[0174] An image feature point recognition module 602 is used to recognize image feature points from the video images to be spliced;

[0175] A spatial feature point matching module 603 is used to match the spatial feature points corresponding to the image feature points according to the image feature points;

[0176] A feature point pair screening module 604 is used to screen the feature point pairs consisting of the image feature points and the space feature points to obtain target feature point pairs;

[0177] An external parameter calculation module 605, used to calculate the external parameters of the camera according to the target feature point pair;

[0178] The video picture stitching module 606 is used to stitch the video pictures to be stitched according to the external parameters of the camera to obtain the stitched target video pictures.

[0179] In an optional embodiment of the present application, the image feature point recognition module 602 includes:

[0180] The dedistortion processing submodule is used to perform dedistortion processing on the video images to be spliced ​​to obtain dedistorted video images;

[0181] A recognition model processing submodule, used to use a recognition model to recognize picture features in the dedistorted video picture; the recognition model is a model trained using a training image containing the picture features;

[0182] The image feature point extraction submodule is used to obtain the image feature points according to the picture features. In an optional embodiment of the present application, the image feature point extraction submodule is also used

[0183] According to the picture features, cropping the dedistorted video picture to obtain a feature thumbnail including the picture features;

[0184] Extracting the image feature points of the feature thumbnail;

[0185] The spatial feature point matching module 603 includes:

[0186] A spatial feature point acquisition submodule, used to acquire the spatial feature points corresponding to the picture features;

[0187] A reference image acquisition submodule, used to acquire a reference image containing reference feature points; there is a first pairing relationship between the coordinate system of the reference image and the spatial coordinate system, and the spatial coordinate system is the coordinate system where the spatial feature points are located;

[0188] A feature corner point matching submodule, used for matching the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point;

[0189] The spatial feature point pairing submodule is used to match the spatial feature points according to the first pairing relationship and the second pairing relationship.

[0190] In an optional embodiment of the present application, the feature point pair screening module 604 includes:

[0191] The iteration number setting submodule is used to set the iteration number;

[0192] The feature point pair filtering submodule is used to filter the noise point pairs in the feature point pairs according to the number of iterations and adopt a random sample consensus algorithm to obtain target feature point pairs.

[0193] In an optional embodiment of the present application, the device further includes:

[0194] A feature point pair classification module, used to classify the target feature point pair to obtain a first feature point pair, a second feature point pair and a third feature point pair;

[0195] A first reprojection error calculation module, used to calculate a first reprojection error according to the first feature point pair, the second feature point pair and the third feature point pair;

[0196] a second reprojection error calculation module, configured to calculate a second reprojection error according to the first feature point pair and the second feature point pair if the first reprojection error is greater than a preset error threshold;

[0197] a third reprojection error calculation module, configured to calculate a third reprojection error according to the first feature point pair if the second reprojection error is greater than the preset error threshold;

[0198] The extrinsic parameter optimization module is used to correct the extrinsic parameter of the camera to the extrinsic parameter calculated based on the previous frame of video if the third reprojection error is greater than the preset error threshold.

[0199] In an optional embodiment of the present application, the external parameters of the camera include a rotation matrix and a translation matrix, and the video picture stitching module 606 includes:

[0200] An affine transformation submodule, used to use the rotation matrix and the translation matrix to perform an affine transformation on the dedistorted video picture to obtain a transformed video picture;

[0201] An overlapping area calculation submodule, used for obtaining the overlapping area of ​​the transformed video picture according to the translation matrix;

[0202] The picture splicing submodule is used to splice the transformed video pictures according to the overlapping area to obtain the spliced ​​target video pictures.

[0203] In an optional embodiment of the present application, applied to a coal mine monitoring system, the video image to be spliced ​​includes a coal mining machine, the coal mining machine includes a body structure and an identification card on the body, the identification card has a unique number, and the image feature point recognition module 602 includes:

[0204] A feature recognition submodule, used to recognize the fuselage structure and the number of the identification card from the video picture to be spliced;

[0205] A feature coordinate acquisition submodule, used to acquire the position coordinates of the fuselage structure and the identification card in the video picture to be spliced;

[0206] The feature point determination submodule is used to identify the image feature points from the video images to be spliced ​​according to the position coordinates.

[0207] In an embodiment of the present application, a video screen to be spliced ​​taken by a camera is obtained; image feature points are identified from the video screen to be spliced; according to the image feature points, spatial feature points corresponding to the image feature points are matched; feature point pairs consisting of image feature points and spatial feature points are screened to obtain target feature point pairs; according to the target feature point pairs, the external parameters of the camera are calculated; according to the external parameters of the camera, the video screen to be spliced ​​is spliced ​​to obtain the target video screen after splicing. The embodiment of the present application performs image feature point identification on the video screen to be spliced, matches the corresponding spatial feature points, screens the feature point pairs consisting of image feature points and spatial feature points, and filters out the coarse noise points generated when extracting the matching feature points, thereby improving the accuracy of the camera external parameter calculation in harsh environments, ensuring the success rate and accuracy of subsequent video screen splicing, and effectively avoiding splicing failures or obvious splicing marks.

[0208] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0209] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0210] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 7 An exemplary apparatus 700 that can be used to implement various embodiments described in this application is schematically illustrated.

[0211] For one embodiment, Figure 7 An exemplary apparatus 700 is shown having one or more processors 702, a control module (chip set) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0212] The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 700 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0213] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage device 708) having instructions 714 and one or more processors 702 combined with the one or more computer-readable media and configured to execute the instructions 714 to implement a module to perform the actions described in the present disclosure.

[0214] For one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 702 and / or any suitable device or component in communication with the control module 704 .

[0215] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0216] The memory 706 may be used, for example, to load and store data and / or instructions 714 for the device 700. For one embodiment, the memory 706 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 706 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0217] For one embodiment, control module 704 may include one or more input / output controllers to provide an interface to NVM / storage device 708 and input / output device(s) 710 .

[0218] For example, NVM / storage 708 may be used to store data and / or instructions 714. NVM / storage 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0219] NVM / storage device 708 may include storage resources that are physically part of the device on which apparatus 700 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 708 may be accessed via input / output device(s) 710 over a network.

[0220] (One or more) input / output devices 710 may provide an interface for the apparatus 700 to communicate with any other appropriate device, and the input / output device 710 may include a communication component, an audio component, a sensor component, etc. The network interface 712 may provide an interface for the apparatus 700 to communicate through one or more networks, and the apparatus 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0221] For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers of the control module 704 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704 to form a system-on-chip (SoC).

[0222] In various embodiments, the apparatus 700 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 700 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0223] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0224] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0225] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0226] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable video screen splicing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable video screen splicing terminal device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0227] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable video image splicing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0228] These computer program instructions can also be loaded onto a computer or other programmable video image splicing terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0229] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0230] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0231] The above is a detailed introduction to a video screen splicing method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video image splicing method, characterized in that: The method comprises: Get the video images to be stitched taken by the camera; Identifying image feature points from the video images to be spliced; According to the image feature points, matching the spatial feature points corresponding to the image feature points; Filtering feature point pairs consisting of the image feature points and the space feature points to obtain target feature point pairs; Calculating the extrinsic parameters of the camera according to the target feature point pair; The video images to be spliced ​​are spliced ​​according to the external parameters of the camera to obtain a spliced ​​target video image.

2. The method according to claim 1, characterized in that The identifying image feature points from the video images to be spliced ​​includes: Performing a dedistortion process on the video images to be spliced ​​to obtain dedistorted video images; Using a recognition model to recognize picture features in the dedistorted video picture; the recognition model is a model trained using a training image containing the picture features; The image feature points are obtained according to the picture features.

3. The method according to claim 2, characterized in that The obtaining the image feature points according to the picture features includes: According to the picture features, cropping the dedistorted video picture to obtain a feature thumbnail including the picture features; Extracting the image feature points of the feature thumbnail; The matching of the spatial feature points corresponding to the image feature points according to the image feature points includes: Acquire the spatial feature point corresponding to the picture feature; Acquire a reference image containing reference feature points; there is a first pairing relationship between the coordinate system of the reference image and the spatial coordinate system, and the spatial coordinate system is the coordinate system where the spatial feature points are located; Matching the feature corner point with the reference feature point to obtain a second pairing relationship between the feature corner point and the reference feature point; The spatial feature points are matched according to the first pairing relationship and the second pairing relationship.

4. The method according to claim 1, characterized in that: The screening of the feature point pairs consisting of the image feature points and the space feature points to obtain target feature point pairs includes: Set the number of iterations; According to the number of iterations, a random sample consensus algorithm is used to filter noise point pairs in the feature point pairs to obtain target feature point pairs.

5. The method according to claim 1, characterized in that After calculating the external parameters of the camera according to the target feature point pair, the method further includes: Classifying the target feature point pairs to obtain a first feature point pair, a second feature point pair, and a third feature point pair; Calculate a first reprojection error based on the first feature point pair, the second feature point pair, and the third feature point pair; If the first reprojection error is greater than a preset error threshold, calculating a second reprojection error according to the first feature point pair and the second feature point pair; If the second reprojection error is greater than the preset error threshold, calculating a third reprojection error based on the first feature point pair; If the third reprojection error is greater than the preset error threshold, the extrinsic parameter of the camera is corrected to the extrinsic parameter calculated according to the previous frame of video.

6. The method according to claim 2, characterized in that The external parameters of the camera include a rotation matrix and a translation matrix. The video images to be spliced ​​are spliced ​​according to the external parameters of the camera to obtain a spliced ​​target video image, including: Using the rotation matrix and the translation matrix, performing an affine transformation on the dedistorted video picture to obtain a transformed video picture; Obtaining an overlapping area of ​​the transformed video picture according to the translation matrix; The transformed video images are spliced ​​according to the overlapping areas to obtain the spliced ​​target video images.

7. The method according to claim 1, characterized in that Applied to a coal mine monitoring system, the video images to be spliced ​​include a coal mining machine, the coal mining machine includes a fuselage structure and an identification card on the fuselage, the identification card has a unique number, and the image feature points are identified from the video images to be spliced, including: Identify the fuselage structure and the number of the identification card from the video picture to be spliced; Obtaining the position coordinates of the fuselage structure and the identification card in the video picture to be spliced ​​respectively; According to the position coordinates, the image feature points are identified from the video images to be spliced.

8. A video image splicing device, characterized in that: The device comprises: The video image acquisition module is used to acquire the video images to be spliced ​​taken by the camera; An image feature point recognition module, used to recognize image feature points from the video images to be spliced; A spatial feature point calculation module, used to match the spatial feature points corresponding to the image feature points according to the image feature points; A feature point pair screening module, used for screening the feature point pairs consisting of the image feature points and the space feature points to obtain target feature point pairs; An extrinsic parameter calculation module, used to calculate the extrinsic parameters of the camera according to the target feature point pair; The video picture splicing module is used to splice the video pictures to be spliced ​​according to the external parameters of the camera to obtain the spliced ​​target video pictures.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, enables the processor to execute the video picture splicing method as described in one or more of claims 1-7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to execute the video picture splicing method as described in one or more of claims 1-7.

Images