Image splicing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an image splicing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a plurality of to-be-spliced images; processing the to-be-spliced images through a preset feature extraction network to obtain high-dimensional feature description vectors corresponding to feature points in the to-be-spliced images; performing feature point matching based on the high-dimensional feature description vectors of the feature points in the to-be-spliced images to obtain similar feature points meeting a preset similar condition; and splicing the to-be-spliced images based on the similar feature points to obtain a spliced image. By adopting the method, the detail features of each image can be obtained, the high-resolution local features are reserved, the matching precision of the feature points is improved based on the extracted high-discrimination feature description vectors, the accuracy of image splicing is ensured, the splicing test effect is good, and the method has wide applicability and flexibility.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image stitching method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Acquiring cabinet images is the basis for realizing digital twin room image processing. In the narrow network environment of the existing network, it is difficult to obtain the global features of the cabinet by photographing the cabinet with a handheld smart terminal. The cabinet is photographed in segments to obtain multiple segmented images, which are usually stitched together manually, resulting in poor image quality of the stitched image. Summary of the invention

[0003] Based on this, it is necessary to provide an image stitching method, device, computer equipment, computer-readable storage medium and computer program product that can improve the image stitching quality and improve the image stitching application scenarios in response to the above technical problems.

[0004] In a first aspect, the present application provides an image stitching method, comprising:

[0005] Obtain multiple images to be stitched;

[0006] Processing each of the images to be spliced ​​by a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to a feature point in each of the images to be spliced;

[0007] Based on the high-dimensional feature description vectors of the feature points in each of the images to be stitched, feature point matching is performed to obtain similar feature points that meet preset similarity conditions;

[0008] The images to be stitched are stitched based on the similar feature points to obtain a stitched image.

[0009] In one embodiment, the step of obtaining a plurality of images to be stitched includes:

[0010] Capturing images of the target object from different shooting angles to obtain a plurality of first images;

[0011] For each of the first images, detect the first image by using a semantic segmentation detection model to obtain mask data of the target object in the first image;

[0012] In the mask data, first pixel points that meet a preset proximity condition with image vertices of the first image are screened out, and contour information of the target object is obtained based on each of the first pixel points;

[0013] The mean value of the angle between the contour information and the horizontal direction is calculated, and the first image is rotated according to the mean value of the angle to obtain an image to be stitched corresponding to the first image.

[0014] In one embodiment, rotating the first image by using the angle mean to obtain the image to be stitched corresponding to the first image includes:

[0015] Rotate the first image by an angle corresponding to the mean of the included angles to obtain a rotationally corrected first image;

[0016] Calculating first perspective transformation parameters based on first position information of each of the first pixels in the first image and second position information of each of the first pixels in the rotationally corrected first image;

[0017] Pixel points other than the first pixel point in the first image are mapped using the first perspective projection transformation parameter to obtain an image to be stitched corresponding to the first image.

[0018] In one embodiment, the processing of each of the images to be stitched by a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to a feature point in each of the images to be stitched includes:

[0019] For each of the images to be stitched, image feature learning is performed on the image to be stitched through a first feature extraction network to obtain a feature response map, and non-maximum suppression processing is performed on the feature response map to obtain at least one stable feature point;

[0020] Inputting each of the stable feature points and the neighborhood image blocks of each of the stable feature points into a second feature extraction network, performing affine transformation learning through the second feature extraction network, and obtaining a mapping relationship between the feature points and the neighborhood image blocks; based on the mapping relationship, obtaining the neighborhood image blocks corresponding to each of the feature points in the feature response map;

[0021] The neighborhood image blocks corresponding to each of the feature points in the feature response graph are input into a third feature extraction network to obtain high-dimensional feature description vectors corresponding to each of the feature points.

[0022] In one embodiment, the high-dimensional feature description vector based on the feature points in each of the images to be stitched is used to match the feature points to obtain similar feature points that meet the preset similarity conditions, including:

[0023] For each feature point to be matched in each of the images to be stitched, based on the high-dimensional feature description vector of the feature point to be matched, a feature distance between the high-dimensional feature description vectors of other images to be stitched is calculated to obtain a feature point whose feature distance satisfies a preset similarity condition;

[0024] The feature points in the other images to be stitched are determined as similar feature points in the other images to be stitched to the feature points to be matched in the images to be stitched.

[0025] In one embodiment, the plurality of images to be stitched include a first image to be stitched and a second image to be stitched; each of the similar feature points is a similar feature point of each first feature point on the first image to be stitched on the second image to be stitched; stitching the images to be stitched based on the similar feature points to obtain a stitched image includes:

[0026] Based on the position information of the feature point in the first image to be stitched, and the position information of the feature point similar to the feature point in the second image to be stitched, obtaining a second perspective transformation parameter;

[0027] Performing perspective transformation on each second feature point on the first image to be stitched based on the second perspective transformation parameter to obtain a transformed first image to be stitched, wherein the transformed first image to be stitched has the same position coordinate system as the second image to be stitched;

[0028] The overlapped pixels in the transformed first image to be stitched and the second image to be stitched are fused to obtain a stitched image of the first image to be stitched and the second image to be stitched.

[0029] In a second aspect, the present application also provides an image stitching method, the method comprising:

[0030] Acquire an image sequence corresponding to the target object, wherein the image sequence includes a plurality of images to be stitched arranged in order of size of image acquisition angles;

[0031] In the image sequence, an adjacent first image to be stitched and a second image to be stitched are obtained, and according to any one of the methods in the first aspect of the embodiments, the adjacent first image to be stitched and the second image to be stitched are processed to obtain a stitching result of the first image to be stitched and the second image to be stitched;

[0032] The image sequence is updated based on the stitching result to obtain an updated image sequence, and based on the updated image sequence, the step of acquiring adjacent first images to be stitched and second images to be stitched in the image sequence is re-executed until a stitched image of the image sequence is obtained.

[0033] In one of the embodiments, updating the image sequence based on the stitching result to obtain an updated image sequence includes:

[0034] The first image to be stitched and the second image to be stitched are replaced by the stitching result to obtain an updated image sequence.

[0035] In one embodiment, the method further comprises:

[0036] The quality evaluation result of the stitched image is obtained through one or more of the aspect ratio of the stitched image, the detection confidence rate of the stitched image, and the image similarity between each of the images to be stitched and the matching area of ​​the stitched image.

[0037] In a third aspect, the present application further provides an image stitching device, the device comprising:

[0038] A first acquisition module is used to acquire multiple images to be stitched;

[0039] A first feature processing module is used to process each of the images to be stitched through a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to a feature point in each of the images to be stitched;

[0040] A second feature processing module is used to perform feature point matching based on the high-dimensional feature description vectors of the feature points in each of the images to be stitched, so as to obtain similar feature points that meet a preset similarity condition;

[0041] The first stitching module is used to stitch the images to be stitched based on the similar feature points to obtain a stitched image.

[0042] In a fourth aspect, the present application further provides an image stitching device, the device comprising:

[0043] A second acquisition module is used to acquire an image sequence corresponding to the target object, wherein the image sequence includes a plurality of images to be stitched arranged in order of size of image acquisition angles;

[0044] a third acquisition module, configured to acquire, in the image sequence, adjacent first images to be stitched and second images to be stitched, and process the adjacent first images to be stitched and second images to be stitched according to the method described in any one of the first aspects to obtain a stitching result of the first images to be stitched and the second images to be stitched;

[0045] An updating module is used to update the image sequence based on the stitching result to obtain an updated image sequence, and based on the updated image sequence, re-execute the step of acquiring adjacent first images to be stitched and second images to be stitched in the image sequence until a stitched image of the image sequence is obtained.

[0046] In a fifth aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above embodiments when executing the computer program.

[0047] In a sixth aspect, the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps described in the above embodiments are implemented.

[0048] In a seventh aspect, the present application also provides a computer program product, including a computer program, which implements the steps described in the above embodiments when executed by a processor.

[0049] The above-mentioned image stitching method, device, computer equipment, computer-readable storage medium and computer program product, wherein the method includes obtaining multiple images to be stitched; processing each of the images to be stitched through a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to the feature points in each of the images to be stitched; matching feature points based on the high-dimensional feature description vectors of the feature points in each of the images to be stitched to obtain similar feature points that meet preset similarity conditions; stitching each of the images to be stitched based on the similar feature points to obtain a stitched image. By adopting this method, the detailed features of each image can be obtained, the high-resolution local features can be retained, the matching accuracy of the feature points can be improved based on the extracted high-discrimination feature description vectors, the accuracy of image stitching can be guaranteed, the stitching test effect is good, and it has wide applicability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 is a schematic diagram of a flow chart of an image stitching method in one embodiment;

[0052] Figure 2 A schematic diagram of a flow chart of a step of obtaining images to be stitched in one embodiment;

[0053] Figure 3 A schematic diagram of a flow chart of a step of obtaining images to be stitched in one embodiment;

[0054] Figure 4 A schematic diagram of a flow chart of a step of obtaining a high-dimensional feature description vector in one embodiment;

[0055] Figure 5 is a flow chart of the steps of obtaining similar feature points in one embodiment;

[0056] Figure 6 A schematic diagram of a process for obtaining a stitched image in one embodiment;

[0057] Figure 7 A schematic diagram of a process for obtaining a stitched image in one embodiment;

[0058] Figure 8 is a schematic diagram of a correction transformation in one embodiment;

[0059] Fig. 9 is a schematic diagram of performing feature point matching in one embodiment;

[0060] Fig.10 is a schematic flow chart of an image stitching method in another embodiment;

[0061] Fig.11 A schematic diagram of splicing the upper figure, the middle figure and the lower figure in one embodiment;

[0062] Fig.12 is a structural block diagram of an image stitching device in one embodiment;

[0063] Fig.13 is a structural block diagram of an image stitching device in one embodiment;

[0064] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] The image stitching method provided in the embodiment of the present application is as follows: Figure 1As shown, an image stitching method is provided, and the execution subject of the image stitching method can be a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the image stitching method can be implemented by a processor calling a computer-readable instruction stored in a memory. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the image stitching method is applied to a terminal device as an example for explanation, and the image stitching method includes the following steps:

[0067] Step 102: Acquire multiple images to be stitched.

[0068] Among them, each image to be stitched may be an image obtained by capturing the target object from different image capture angles, each image to be stitched may also be an image obtained by capturing the target object from different shooting heights of the target object, and the image to be stitched may also be an image obtained by performing the image stitching method described in this embodiment on two images to be stitched. The target object may be an environment, an object, etc., different image capture angles may be different image shooting angles, and capturing the image of the target object may be photographing the target object. In one example, the terminal device may photograph the target object to obtain an upper partial image, a middle partial image, a lower partial image, and the like of the target object.

[0069] Specifically, the terminal device can receive multiple images to be stitched sent by other devices, and the terminal device can also capture images of the target object through an image acquisition device to obtain multiple images to be stitched; for example, the target object can be a cabinet, and the multiple images to be stitched obtained by the terminal device can be an upper cabinet image obtained by photographing the cabinet at a first height, a middle cabinet image obtained by photographing the cabinet at a second height, and a lower cabinet image obtained by photographing the cabinet at a third height.

[0070] Step 104 , processing each image to be spliced ​​by a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to a feature point in each image to be spliced.

[0071] Among them, the preset feature extraction network may include multiple feature extraction sub-networks for extracting multi-level feature vectors. The preset feature extraction network may include multiple convolutional neural networks, for example, may include a first convolutional neural network, a second convolutional neural network and a third convolutional neural network. The first convolutional neural network may be KeyNet, the second convolutional neural network may be AffNet, and the third convolutional neural network may be HardNet. The size information of the convolution kernel of each convolutional neural network may be determined based on the actual application scenario; the feature point may be determined based on the output result of Keynet, and the high-dimensional feature description vector may be a descriptor of the target dimension, for example, a 128-dimensional descriptor.

[0072] Specifically, for each image to be stitched, the terminal device can input the image to be stitched into a preset feature extraction network, process the image to be stitched through the preset feature extraction network, and obtain high-dimensional feature description vectors corresponding to the feature points of the image to be stitched.

[0073] Step 106 , performing feature point matching based on the high-dimensional feature description vectors of the feature points in each of the images to be stitched, to obtain similar feature points that meet a preset similarity condition.

[0074] Among them, the feature point can be each feature point in the image to be stitched, and the high-dimensional feature description vector can be obtained by extracting features from the neighborhood image blocks corresponding to each feature point; feature point matching can be a process of matching based on the similarity between the high-dimensional feature description vectors of each feature point in different images to be stitched; the content of the preset similarity condition can be that the distance between the high-dimensional feature description vectors of two feature points is less than a preset distance threshold, or that the high-dimensional feature description vectors between the two feature points are similar or identical, etc.; similar feature points are for the feature points to be matched in the image to be stitched, and the similar feature points can be feature points that represent that the high-dimensional feature description vectors of the feature points to be matched on other images to be stitched except the image to be stitched meet the preset similarity conditions.

[0075] Step 108 , stitching the images to be stitched based on the similar feature points to obtain a stitched image.

[0076] Specifically, the terminal device can perform image stitching based on each feature point to be matched on the image to be stitched and similar feature points of each feature point to be matched on other images to be stitched, to obtain a stitched image of the image to be stitched and the other images to be stitched. For example, the terminal can merge the pixel points of the feature points to be matched on the image to be stitched and the pixel points of similar feature points of each feature point to be matched on other images to be stitched, to obtain a stitched image between the image to be stitched and the other images to be stitched.

[0077] In the above-mentioned image stitching method, a plurality of images to be stitched are obtained; each of the images to be stitched is processed through a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to the feature points in each of the images to be stitched; based on the high-dimensional feature description vectors of the feature points in each of the images to be stitched, feature point matching is performed to obtain similar feature points that meet preset similarity conditions; based on each of the similar feature points, each of the images to be stitched is stitched to obtain a stitched image. By adopting this method, the detailed features of each image can be obtained, the high-resolution local features can be retained, the matching accuracy of the feature points can be improved based on the extracted high-discrimination feature description vectors, the accuracy of image stitching can be ensured, the stitching test effect is good, and it has wide applicability and flexibility.

[0078] In an exemplary embodiment, Figure 2 As shown, the specific implementation process of the step of "obtaining multiple images to be stitched" may include:

[0079] Step 202: for each of the multiple first images, detect the first image using a semantic segmentation detection model to obtain mask data of the target object in the first image.

[0080] Among them, each of the first images may be an image obtained by capturing images of the target object from different shooting angles, each of the first images may also be an image obtained by capturing images of the target object from different shooting heights of the target object, and the first image may also be an image obtained by performing the image stitching method described in the above embodiment on two images to be stitched. The semantic segmentation detection model may be used to detect the target object in the first image, and the semantic segmentation detection model may be a pre-trained neural network; the mask data of the target object may be the mask data in the output result of the semantic segmentation detection model.

[0081] Specifically, for each of the multiple first images acquired by the terminal device, the image stitching method described in this embodiment can be executed, and the terminal device can perform semantic detection on the first image through a semantic segmentation detection model, and semantically classify each pixel in the first image to obtain a semantic detection result; the semantic segmentation detection model can be used to identify the target object in the first image, and the output semantic segmentation result can include the identification of each pixel in the first image, and the identification of each pixel is used to characterize whether the pixel belongs to the target object. The terminal device can determine each pixel belonging to the target object based on the identification of each pixel in the first image, and determine the pixel belonging to the target object as mask data.

[0082] In one example, the identifier of each pixel point in the first image output as a result of the semantic segmentation may include a first identifier and a second identifier, wherein the first identifier may be used to characterize that the pixel point belongs to the target object, and the second identifier may be used to characterize that the pixel point does not belong to the target object; for example, the first identifier may be 1, and the corresponding second identifier may be 0, and so on. The present disclosure does not limit the specific implementation method of the first identifier and the second identifier.

[0083] Step 204 : Filter out first pixel points that meet a preset proximity condition with the image vertices of the first image in the mask data, and obtain contour information of the target object based on each first pixel point.

[0084] The preset proximity condition may be a pixel point that is closest to an image vertex, and the first pixel point that satisfies the preset proximity condition in the first image may be a pixel point that is at the shortest distance from an image vertex of the first image.

[0085] Specifically, the terminal device can obtain the output result of the semantic segmentation model, and determine the pixel points belonging to the target object in the first image as mask data based on the output result; the terminal device can determine the image vertices of the first image; among the multiple pixel points contained in the mask data, the terminal device can screen based on the distance between each pixel point and the image vertex of the first image, respectively determine the pixel points closest to each image vertex, and determine the pixel point closest to each image vertex as the first pixel point. The terminal can obtain the contour information of the first image based on each first pixel point.

[0086] In one example, the specific process of the terminal device determining the contour information may include: the terminal device may determine each first pixel point as a vertex of the target object in the first image, and obtain the contour information of the target object in the first image based on the pixel points at the edge position and each pixel point in the mask data. The target object may be a server cabinet, and the first image may be an image captured by the server cabinet, and the first image includes the server cabinet.

[0087] Step 206 , calculating the mean value of the angle between the contour information and the horizontal direction, and performing rotation processing on the first image according to the mean value of the angle to obtain the image to be stitched corresponding to the first image.

[0088] Among them, the mean value of the angle between the contour information and the horizontal direction can be the mean value of the angle between the contour data of the target object in the first image and the horizontal standard direction; the mean value of the angle between the contour information and the horizontal direction can be calculated based on the angle between the first contour data in the horizontal direction of the target object and the horizontal direction, and the angle between the second contour data in the horizontal direction of the target object and the horizontal direction.

[0089] Specifically, based on the contour information of the target object in the first image, the terminal device can obtain the first contour data and the second contour data of the target object in the horizontal direction, and can calculate the first angle between the first contour data in the horizontal direction and the horizontal standard direction, and the second angle between the second contour data in the horizontal direction and the horizontal standard direction. Based on this, the terminal device can calculate the mean of the first angle and the second angle to obtain the angle mean. The terminal device can rotate the first image by an angle corresponding to the angle mean to obtain the image to be spliced ​​corresponding to the first image. The first contour data can be the first side of the target object in the horizontal direction and the second side of the target object in the horizontal direction.

[0090] In this embodiment, the target object in the first image can be screened, and the target object can be rotationally corrected based on the pixel points corresponding to the screened target object. The image to be stitched can be preprocessed, effectively improving the accuracy of image registration.

[0091] In an exemplary embodiment, Figure 3 As shown, the specific implementation process of the step of "rotating the first image by the angle mean to obtain the image to be stitched corresponding to the first image" may include:

[0092] Step 302: Rotate the first image by an angle corresponding to the mean of the included angles to obtain a rotationally corrected first image.

[0093] Specifically, the terminal device can perform rotation correction on the first image based on the angle corresponding to the mean of the angles to obtain the first image after rotation correction, that is, the terminal device can rotate the first image by the angle corresponding to the mean of the angles to obtain the first image after rotation correction. In one example, the terminal device can rotate the pixel points in the first image belonging to the target image by the angle corresponding to the mean of the angles to obtain the target object image after rotation correction.

[0094] Step 304 : Calculate first perspective transformation parameters based on first position information of each first pixel in the first image and second position information of each first pixel in the first image after rotation correction.

[0095] Among them, the first position information of the first pixel point can be the position coordinates of the first pixel point, that is, the first position information is the coordinate information of the first pixel point on the first image; the second position information is the coordinate information of the first pixel point on the first image after rotation correction, that is, the second position information is the coordinate information of the first pixel point on the target object image after rotation correction.

[0096] Specifically, the terminal device can determine the coordinate information of each first pixel in the first image, that is, the first position information of each first pixel, and determine the coordinate information of each first pixel on the target object image after rotation correction, that is, the second position information, and perform calculations based on the first position information and the second position information of each first pixel to obtain the first perspective transformation parameters, that is, calculate the first perspective transformation parameters for transforming the first position information into the second position information.

[0097] In an example, the first position information of the first pixel may be (x1, y1, 1), and the corresponding second position information of the first pixel may be (x1', y1', 1). The perspective transformation between the first position information and the second position information of the first pixel may be performed by the following formula:

[0098] t ij ∈T.

[0099] in, It can be the first perspective transformation parameter. T is the homography matrix.

[0100] Step 306 , mapping the pixel points in the first image except the first pixel point by using the first perspective projection transformation parameter to obtain the image to be stitched corresponding to the first image.

[0101] Specifically, the terminal device can transform the position information of other pixel points on the first image except the first pixel point based on the calculated first perspective transformation parameter, obtain the position information of each other pixel point after the perspective transformation, and adjust each pixel point based on the position information of each pixel point after the perspective transformation, so as to obtain the image to be stitched corresponding to the first image.

[0102] In one example, for each pixel point other than the first pixel point on the first image, the terminal device can obtain the position information of the other pixel point, calculate the quotient between the position information and the first perspective transformation parameter, and determine the quotient as the position information after the perspective transformation corresponding to the other pixel point, and adjust each pixel point based on the position information after the perspective transformation corresponding to each other pixel point to obtain the image to be stitched of the first image.

[0103] In another example, after the terminal device adjusts each pixel based on the position information of each other pixel after perspective transformation to obtain the image to be stitched of the first image, it can also supplement the pixel values ​​of the missing pixels in the image to be stitched of the first image based on a preset interpolation method to obtain a complete image to be stitched of the first image.

[0104] In this embodiment, the first image may be pre-processed for perspective correction, which provides a stable data basis for improving the accuracy and reliability of image stitching in subsequent processes.

[0105] In an exemplary embodiment, Figure 4 As shown, the specific implementation process of the step of "processing each image to be spliced ​​by a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to the feature points in each image to be spliced" may include:

[0106] Step 402: for each image to be stitched, perform image feature learning on the image to be stitched through a first feature extraction network to obtain a feature response map, perform non-maximum suppression processing on the feature response map to obtain at least one stable feature point.

[0107] Among them, the preset feature extraction network may include multiple feature extraction sub-networks for extracting multi-level feature vectors. The preset feature extraction network may include multiple convolutional neural networks, for example, may include a first convolutional neural network, a second convolutional neural network and a third convolutional neural network. The first convolutional neural network may be KeyNet, the second convolutional neural network may be AffNet, and the third convolutional neural network may be HardNet.

[0108] Specifically, for each image to be spliced, the terminal device can input the image to be spliced ​​into the first feature extraction network in the preset feature extraction network, and learn the image features of the image to be spliced ​​through the convolutional neural network in the first feature extraction network to obtain the feature response map of the image to be spliced ​​output by the first feature extraction network; the terminal device can perform non-maximum suppression processing on the feature response map through a preset non-maximum suppression algorithm, determine the local maximum value in the feature response map, and determine the feature points corresponding to each local maximum value output by the NMS algorithm as stable feature points. In other words, the feature response map corresponding to the image to be spliced ​​contains multiple feature points, and the terminal device can perform non-maximum suppression processing on the feature response map, screen the multiple feature points contained in the feature response map, and determine the feature points corresponding to the local maximum value output by the NMS algorithm as stable feature points of the image to be spliced, that is, the terminal device can obtain one or more stable feature points of the image to be spliced.

[0109] Step 404: Input each stable feature point and the neighborhood image block of each stable feature point into the second feature extraction network, perform affine transformation learning through the second feature extraction network, and obtain the mapping relationship between the feature point and the neighborhood image block. Based on the mapping relationship, obtain the neighborhood image block corresponding to each feature point in the feature response map.

[0110] The neighborhood image block of the stable feature point may be composed of pixel points adjacent to the pixel point of the stable feature point.

[0111] Specifically, after obtaining each stable feature point, the terminal device can respectively determine the neighborhood image blocks composed of adjacent pixel points of each stable feature point, and input each stable feature point and the neighborhood image blocks corresponding to each stable feature point into the second feature extraction network. The second feature extraction network can learn affine transformation from each stable feature point and the neighborhood image blocks corresponding to each stable feature point through the convolutional neural network contained therein, and obtain the mapping relationship between each stable feature point and the neighborhood image block. Based on this, the terminal device can process each feature point in the feature response map based on the obtained mapping relationship to generate the neighborhood image blocks corresponding to each feature point. The mapping relationship describes the local shape of each feature point.

[0112] Step 406: input the neighborhood image blocks corresponding to each feature point in the feature response image into a third feature extraction network to obtain a high-dimensional feature description vector corresponding to each feature point.

[0113] Specifically, the terminal device can input the neighborhood image blocks corresponding to each feature point of the image to be stitched into the third feature extraction network, and extract features from the image blocks through the convolutional neural network in the third feature extraction network to obtain high-dimensional descriptors corresponding to each feature point, that is, high-dimensional feature description vectors. In an example, the output of the third feature extraction network can be 128-dimensional feature description vectors corresponding to each feature point in the image to be stitched, that is, 128-dimensional descriptors.

[0114] In this embodiment, a feature extraction network obtained by combining KeyNet, AffNet and HardNet is used to perform image detection on the stitched image. KeyNet realizes stable and consistent key point detection for the stitched image, and can maintain a high key point repetition rate when the image is rotated, scaled and illuminated. AffNet is invariant to viewing angle changes and perspective deformation. The high-resolution feature descriptor output by HardNet can further improve the matching accuracy between feature points.

[0115] In one embodiment, Figure 5 As shown, the specific implementation process of the step of "matching feature points based on the high-dimensional feature description vectors of the feature points in each image to be stitched to obtain similar feature points that meet the preset similarity conditions" may include:

[0116] Step 502, for each feature point to be matched in each image to be stitched, based on the high-dimensional feature description vector of the feature point to be matched, calculate the feature distance between the feature point and the high-dimensional feature description vectors of other images to be stitched, and obtain feature points whose feature distances meet preset similarity conditions.

[0117] The content of the preset similarity condition may be that the feature distance is the smallest, the content of the preset similarity condition may also be that the similarity between high-dimensional feature description vectors is the highest, and so on.

[0118] Specifically, the terminal device can obtain each feature point to be matched in each image to be stitched, obtain a high-dimensional feature description vector of the feature point to be matched, and screen between the high-dimensional feature description vectors corresponding to each feature point in other images to be stitched to obtain a feature point that meets a preset similarity condition with the feature point to be matched.

[0119] In one example, the image to be stitched may be the first image to be stitched, and the other image to be stitched may be the second image to be stitched; the first image to be stitched may contain multiple feature points to be matched, and for each feature point to be matched contained in the first image to be stitched, the terminal device may perform feature point matching through the AdaLAM algorithm, obtain a high-dimensional feature description vector of the feature point to be matched, and screen based on the high-dimensional feature description vectors corresponding to each feature point in the second image to be stitched, calculate the feature distance between the high-dimensional feature description vector of the feature point to be matched and the high-dimensional feature description vector of each feature point in the second image to be stitched, and determine the feature point corresponding to the high-dimensional feature description vector with the smallest feature distance as the feature point of the feature point to be matched that meets the preset similarity condition.

[0120] In another example, the terminal device can calculate the similarity between the high-dimensional feature description vector of the feature point to be matched and the high-dimensional feature description vector of each feature point in the second image to be stitched, and determine the feature point corresponding to the high-dimensional feature description vector with the greatest similarity as the feature point of the feature point to be matched that meets the preset similarity condition.

[0121] Step 504 : determining feature points in other images to be stitched, which are similar feature points of the feature points to be matched in the image to be stitched in other images to be stitched.

[0122] Specifically, the terminal may determine the feature points in other images to be stitched that meet the preset similarity condition as similar feature points of the feature points to be matched in the image to be stitched in the other images to be stitched. That is, the image to be stitched may be the first image to be stitched, and the other images to be stitched may be the second image to be stitched; the terminal device may determine the feature points in the second image to be stitched that meet the preset similarity condition as the feature points of the feature points to be matched in the second image to be stitched.

[0123] In one example, the terminal device may also use the RANSAC algorithm to eliminate mismatched points, further ensuring high accuracy of feature point matching.

[0124] In this embodiment, the high-dimensional feature description vector of the feature point is used as a reference for matching between the feature points, which can be guaranteed.

[0125] In one embodiment, the plurality of images to be stitched include a first image to be stitched and a second image to be stitched, and each similar feature point is a similar feature point of each first feature point on the first image to be stitched on the second image to be stitched.

[0126] Accordingly, if Figure 6 As shown, the specific implementation process of the step of "stitching the images to be stitched based on the similar feature points to obtain the stitched image" may include:

[0127] Step 602 : obtaining second perspective transformation parameters based on the position information of the feature point in the first image to be stitched and the position information of the feature point similar to the feature point in the second image to be stitched.

[0128] The terminal device may obtain the second perspective transformation parameter by performing calculation based on the position information of the feature point in the first image to be stitched and the position information of a similar feature point to the feature point in the second image to be stitched.

[0129] In an example, the position information of the feature point in the first image to be stitched may be (x1, y1, 1), and the position information of the similar feature point of the corresponding feature point in the second image to be stitched may be (x1', y1', 1), and the perspective transformation may be performed by the following formula:

[0130] t ij ∈T.

[0131] in, It can be the second perspective transformation parameter. T is the homography matrix.

[0132] Step 604 : Perform perspective transformation on each second feature point on the first image to be stitched based on the second perspective transformation parameter to obtain a transformed first image to be stitched.

[0133] The transformed first image to be stitched has the same position coordinate system as the second image to be stitched. Other feature points on the first image to be stitched except the first feature point may be the second feature point.

[0134] Specifically, the terminal device can transform the position information of the pixel points of the second feature point based on the calculated second perspective transformation parameters, obtain the position information of the pixel points corresponding to the second feature point after the perspective transformation, and adjust the position of the pixel points of each second feature point based on the position information after the perspective transformation to obtain the transformed first image to be spliced.

[0135] In one example, for each pixel point of the second feature point, the terminal device can obtain the position information of the pixel point of the second feature point, calculate the quotient between the position information and the second perspective transformation parameter, and determine the quotient as the position information of the pixel point of the second feature point after the perspective transformation, and adjust the pixel points of each second feature point based on the position information after the perspective transformation to obtain the transformed first image to be spliced.

[0136] Step 606 , performing fusion processing on the overlapped pixels in the transformed first image to be stitched and the second image to be stitched, to obtain a stitched image of the first image to be stitched and the second image to be stitched.

[0137] Specifically, the terminal device can merge the transformed overlapping pixels in the first image to be stitched and the second image to be stitched to obtain a stitched image of the first image to be stitched and the second image to be stitched. In an example, the terminal device can also fuse the overlapping pixels in the first image to be stitched and the second image to be stitched, for example, by using an Alpha algorithm to avoid discontinuity at the stitching.

[0138] In this embodiment, the first image to be stitched and the second image to be stitched can be transformed to the same plane by performing perspective transformation, thereby improving the quality of the stitched image.

[0139] The image stitching method provided in the embodiment of the present application is as follows: Figure 7 As shown, an image stitching method is provided. The execution subject of the image stitching method can be a terminal device or a server or other processing device. The image stitching method is applied to a terminal device as an example for description. The image stitching method includes the following steps:

[0140] Step 702: Acquire an image sequence corresponding to the target object.

[0141] Specifically, the image sequence includes a plurality of images to be stitched arranged in order of image acquisition angles. In one example, the image sequence may also include a plurality of images to be stitched arranged in order of image acquisition heights.

[0142] Step 704: in the image sequence, obtain adjacent first images to be stitched and second images to be stitched, and process the adjacent first images to be stitched and second images to be stitched according to the method of any of the above embodiments to obtain a stitching result of the first images to be stitched and the second images to be stitched.

[0143] Specifically, the terminal device can obtain two adjacent images to be stitched in the image sequence in order of arrangement, for example, the first image to be stitched and the second image to be stitched. The terminal device can perform the implementation process of the image stitching method described in the above embodiment based on the acquired first image to be stitched and the second image to be stitched, obtain a stitched image of the first image to be stitched and the second image to be stitched, and determine the stitched image as the stitching result of the first image to be stitched and the second image to be stitched.

[0144] Step 706, updating the image sequence based on the stitching result to obtain an updated image sequence, and re-executing the step of acquiring adjacent first images to be stitched and second images to be stitched in the image sequence based on the updated image sequence, until a stitched image of the image sequence is obtained.

[0145] Specifically, the terminal device can update the image sequence based on the stitching result to obtain an updated image sequence, and re-execute the step of acquiring adjacent first images to be stitched and second images to be stitched in the updated image sequence to obtain the stitching result corresponding to the step, until the updated image sequence contains only one image, and the image is determined as the stitched image of the image sequence.

[0146] In one example, the terminal device can obtain the first two images to be stitched in the image sequence according to the arrangement order of the images, determine them as the first image to be stitched and the second image to be stitched, execute the method described in the above embodiment to obtain a stitched image of the first image to be stitched and the second image to be stitched, that is, a stitching result, and replace the first image to be stitched and the second image to be stitched in the image sequence with the stitching result to obtain an updated image sequence, at which time the number of images to be stitched included in the updated image sequence is one less than the number of images to be stitched included in the image sequence before the update; based on this, the terminal device can re-determine the first image to be stitched and the second image to be stitched from the first two images to be stitched in the updated image sequence, execute the method described in the above embodiment to obtain a stitching result, and replace the first image to be stitched and the second image to be stitched with the stitching result to obtain an updated image sequence, until the number of images to be stitched in the updated image sequence is 1, then the image currently included in the updated image sequence can be determined as the stitched image of the image sequence.

[0147] In a specific example, the image sequence may include image 1 to be stitched, image 2 to be stitched, and image 3 to be stitched, which are arranged in sequence; the terminal device may determine image 1 to be stitched as the first image to be stitched, and image 2 to be stitched as the second image to be stitched, to obtain a stitching result of image 1 to be stitched and image 2 to be stitched, which may be image a, for example; the terminal device may update the image sequence based on image a, and replace image 1 to be stitched and image 2 to be stitched with image a, that is, the updated image sequence may include image a arranged in sequence and image 3 to be stitched. At this time, the terminal device can re-execute the method described in the above embodiment based on the updated image sequence, determine image a as the first image to be stitched, determine image 3 to be stitched as the second image to be stitched, and obtain the stitching result of image a and image 3 to be stitched, that is, image b; and replace image a and image 3 to be stitched in the image sequence with the stitching result (image b). The updated image sequence only contains image b, and the terminal device can determine that image b is the stitched image of image 1 to be stitched, image 2 to be stitched, and image 3 to be stitched.

[0148] In another specific example, the image sequence may include image 1 to be stitched, image 2 to be stitched, and image 3 to be stitched arranged in sequence; the terminal device may determine image 2 to be stitched as a reference image, and determine the reference image as the second image to be stitched; the terminal device may determine image 1 to be stitched as the first image to be stitched, and obtain a first stitching result of image 1 to be stitched and image 2 to be stitched; the terminal device may also determine image 3 to be stitched as the first image to be stitched, and obtain a second stitching result of image 3 to be stitched and image 2 to be stitched; based on the first stitching result and the second stitching result, a stitched image of the image sequence is obtained.

[0149] In one example, the number of images to be stitched included in the image sequence may be an even number. Accordingly, the process of the terminal device acquiring the adjacent first image to be stitched and the second image to be stitched in the image sequence may be to respectively acquire multiple groups of adjacent image groups to be stitched according to the arrangement order of the images in the image sequence, and obtain the stitching results of the image groups to be stitched; and determine the stitching results of the image groups to be stitched as the updated image sequence. If the terminal device determines that the number of images included in the updated image sequence is an even number, multiple groups of adjacent image groups to be stitched may be acquired again according to the arrangement order of the images in the image sequence, and obtain the stitching results of the image groups to be stitched, until the number of images included in the updated image sequence is 1, and the images included in the image sequence at this time are determined as the stitched images of the image sequence; if the terminal device determines that the number of images included in the updated image sequence is an odd number, the first two images in the image sequence may be determined as the first image to be stitched and the second image to be stitched, and the stitching results may be obtained, until the number of images included in the updated image sequence is 1, and the images included in the image sequence at this time are determined as the stitched images of the image sequence. There are many ways for the terminal device to determine the adjacent first image to be stitched and the second image to be stitched, which will not be described in detail here.

[0150] In the above-mentioned image stitching method, by matching feature points of a plurality of images to be stitched contained in an image sequence, the stitching success rate can be improved while ensuring the stitching efficiency.

[0151] In an exemplary embodiment, the specific implementation process of the step of “updating the image sequence based on the stitching result to obtain an updated image sequence” may include:

[0152] The first image to be stitched and the second image to be stitched are replaced by the stitching result to obtain an updated image sequence.

[0153] Specifically, the terminal device can obtain the first two images to be stitched in the image sequence according to the arrangement order of the images, determine them as the first image to be stitched and the second image to be stitched, execute the method described in the above embodiment, obtain a stitched image of the first image to be stitched and the second image to be stitched, that is, a stitching result, and replace the first image to be stitched and the second image to be stitched in the image sequence with the stitching result to obtain an updated image sequence. At this time, the number of images to be stitched contained in the updated image sequence is one less than the number of images to be stitched contained in the image sequence before the update.

[0154] In this embodiment, the stitching of multiple images to be stitched is achieved by continuously updating the image sequence.

[0155] In one embodiment, the image stitching method further includes:

[0156] The quality evaluation result of the stitched image is obtained through one or more of the aspect ratio of the stitched image, the detection confidence rate of the stitched image, and the image similarity between each image to be stitched and the matching area of ​​the stitched image.

[0157] Specifically, the terminal device can obtain a quality evaluation result of the stitched image by determining one or more of the aspect ratio of the stitched image, the detection confidence rate of the stitched image, and the image similarity between each image to be stitched and the matching area of ​​the stitched image. The quality evaluation result may include a stitching success result, or may also include a stitching failure result.

[0158] In an example, the terminal device may determine the quality evaluation result of the stitched image according to the aspect ratio of the stitched image, the detection confidence rate of the stitched image, and the image similarity between each image to be stitched and the matching area of ​​the stitched image.

[0159] In one example, the terminal device first performs a ratio detection, and can determine a reference image in the image sequence, for example, the image to be spliced ​​that is located first in the arrangement order in the image sequence can be determined as the reference image, or the image to be spliced ​​that is located in the middle of the arrangement order can be determined as the reference image; the terminal device can obtain the aspect ratio of the reference image and the aspect ratio of the spliced ​​image, and if the aspect ratio of the spliced ​​image does not change compared with the aspect ratio of the reference image, the terminal device can determine that the quality evaluation result corresponding to the spliced ​​image is a splicing failure result. If the aspect ratio of the spliced ​​image does not change compared with the aspect ratio of the reference image, the terminal device can determine that the spliced ​​image passes the ratio detection.

[0160] The terminal device can perform model training to obtain a target object detection model, namely, a server cabinet detection model, and the terminal device can perform target detection through the server cabinet detection model; if the terminal device determines that the spliced ​​image passes the ratio detection, the terminal device can perform cabinet target detection on the spliced ​​image through the server cabinet detection model, and the result output by the server cabinet detection model can be the detection confidence rate of the spliced ​​image; if the terminal device determines that the currently output detection confidence rate is less than a preset confidence rate threshold, the quality evaluation result corresponding to the spliced ​​image of the terminal device is a splicing failure result. If the currently output detection confidence rate is greater than or equal to the preset confidence rate threshold, the terminal device can determine that the spliced ​​image passes the target detection.

[0161] If it is determined that the stitched image passes the target detection, the terminal device can detect the feature points in the stitched image through the preset feature extraction network in the above embodiment, and determine the matching areas in the stitched image with all the images to be stitched except the reference image through AdaLAM feature matching. The structural similarity (Structural Similarity Index, SSIM) is used to compare the similarity between the image to be stitched and the matching area of ​​the stitched image. If the similarity (i.e., the structural similarity index) between the matching area of ​​the image to be stitched and the stitched image except the reference image is greater than the preset similarity threshold, the quality evaluation result of the stitched image is determined to be a successful stitching result. The preset similarity threshold and the preset confidence rate threshold can be determined based on specific values ​​of the actual application scenario, and the present disclosure does not specifically limit the specific values.

[0162] In this embodiment, the quality of the stitched image can be comprehensively detected in multiple dimensions through ratio detection, target detection, and similarity detection to avoid the occurrence of stitched images with low stitching quality and provide a reliable data basis for subsequent image processing.

[0163] In the following, in conjunction with a specific embodiment, a detailed description of the specific implementation process of the above-mentioned image stitching method may include:

[0164] In the current network, the equipment in the computer room is frequently taken off the shelves, and the audit and maintenance work rely on manual verification, which not only consumes a lot of manpower and material resources, but also makes it difficult for the update of resource system data to keep up with the equipment update in the computer room. By combining images to obtain computer room information, the digital twin creates a computer room corresponding to reality in the digital space, and provides more efficient management and maintenance means by simulating and analyzing the computer room in the digital space. The image stitching method provided in this embodiment can avoid the problem of handheld devices taking pictures of the whole picture of the cabinet in the narrow environment of the current network. Image stitching is achieved through image correction, feature extraction of KeyNet, AffNet and HardNet combination, and AdaLAM feature matching, and automatic image stitching quality evaluation is achieved based on adaptive feature matching, which provides a complete and high-resolution cabinet image for subsequent image-based digital computer room processing such as equipment model recognition, text recognition and computer room digital modeling.

[0165] Acquiring cabinet images is the basis for realizing image processing of digital twin computer rooms. In the narrow environment of the existing network, it is difficult to obtain the global features of the cabinet by photographing the cabinet with a handheld smart terminal, and the resolution of a single image is not enough to extract clear detail features, such as the cabinet QR code, terminal ports, and text on labels. To address this problem, the image stitching method provided in this embodiment proposes an image stitching method based on adaptive feature matching to construct a complete cabinet image while retaining high-resolution local features as much as possible. The stitched image of the cabinet obtained by the image stitching method provided in this embodiment can be used as the basis for subsequent digital twin computer rooms. In the 100 sets of cabinet image stitching tests, an 82% success rate was achieved, which is a high success rate. At the same time, it can be extended to different application scenarios and different stitching target objects without pre-training, and has wide applicability and flexibility.

[0166] Specifically, the image stitching method provided in this embodiment can first shoot the upper, middle and lower sections of the cabinet, and perform rotation correction and perspective transformation correction on the three images to be stitched; secondly, the feature extractor of the combination of KeyNet, AffNet and HardNet extracts high-dimensional feature descriptors from the image; and uses AdaLAM feature matching to find accurate matching points between images; finally, the image stitching is achieved by solving the homography matrix through random sampling consistency (RANdom SAmple Consensus, RANSAC). In addition, the image stitching method provided in this embodiment can also realize automatic stitching quality evaluation, and realize comprehensive evaluation of stitching quality through the aspect ratio of the stitched image, the cabinet target detection confidence rate, and the image similarity between the matching areas of the image to be stitched and the stitching result image, so as to ensure the accuracy and reliability of the stitching result.

[0167] In the image stitching method provided in this embodiment, Digital Twin refers to creating a virtual model corresponding to a real physical object or system in a virtual space by digital means. This virtual model is not just a simple three-dimensional geometric figure, but contains all the characteristics and behaviors of the physical object or system, and can be simulated, analyzed and optimized in a virtual environment.

[0168] Semantic segmentation is a task in computer vision, whose goal is to classify each pixel in an image into a specific category. Unlike image classification tasks, semantic segmentation not only determines which categories exist in the image, but also accurately locates the locations of these categories in the image. In other words, semantic segmentation requires labeling each pixel in the image so that each pixel corresponds to a specific category label.

[0169] Image stitching refers to the seamless synthesis of multiple partially overlapping images or video frames into a complete large-size image or video through computer algorithms and technologies.

[0170] Random Sample Consensus (RANSAC) is a classic robust estimation algorithm used to estimate the parameters of a mathematical model from a set of data containing noise and outliers. The basic idea is to estimate the model parameters by randomly selecting a subset of data (called samples or inliers), and determine the optimal model by evaluating the fitness of the model on these inliers.

[0171] Structural Similarity Index (SSIM) is a metric used to measure the similarity between two images. It is different from traditional pixel difference metrics (such as mean square error), but is based on the perceptual characteristics of the human visual system and takes into account the effects of brightness, contrast, and structural information.

[0172] Specifically, the image stitching method provided in this embodiment includes the process of image rotation correction and perspective correction based on semantic segmentation:

[0173] 1. Use the semantic segmentation detection model to detect the cabinet in the original image, obtain the cabinet mask data, and extract the four points closest to the vertices of the original image in the cabinet mask data as the cabinet vertices to construct the cabinet contour, that is, obtain the contour information of the target object.

[0174] 2. Image rotation correction: Figure 8 As shown, input the original image, such as Figure 8 The left side of the image containing the target object is calculated by calculating the average of the angles between the two short sides of the cabinet outline and the horizontal axis, and the image is rotated by a corresponding angle to achieve rotation correction of the image, thereby obtaining a rotation corrected image.

[0175] 3. Perspective transformation correction: Input the image after rotation correction and use the cabinet vertex as the correction positioning point, that is, Figure 8 For the coordinate point (x1, y1, 1) on the two-dimensional image (original image) and the corresponding coordinate point (x1', y1', 1) on the target image (image corrected by perspective transformation), the perspective transformation is

[0176] t ij ∈T.

[0177] in, It can be the first perspective transformation parameter. T is the homography matrix.

[0178] Substitute the extracted correction positioning points into the solution of the homography matrix T to obtain the first perspective transformation parameters. Finally, each pixel in the original image is mapped through the homography matrix to obtain the corresponding pixel on the target image (corrected image), and the gaps are filled in according to the interpolation method to achieve image correction.

[0179] Specifically, the image stitching method provided in this embodiment also includes a feature extraction process of a combination of KeyNet, AffNet and HardNet:

[0180] In view of the complex environment of the computer room, messy wiring and poor lighting, the image stitching method provided in this embodiment innovatively uses a combination of KeyNet, AffNet and HardNet networks to extract feature points for image stitching. KeyNet provides stable and consistent key point detection, and AffNet is invariant to changes in viewing angles and perspective deformation. On this basis, HardNet generates a high-discrimination feature descriptor to improve matching accuracy. The combination of the three realizes multi-level feature extraction suitable for complex scenes. The specific steps are as follows:

[0181] 1. KeyNet: Input the rectified image, KeyNet learns the image features through the convolutional neural network (CNN), and finally obtains the feature response map corresponding to the rectified image. By performing non-maximum suppression on the feature response map corresponding to the rectified image, stable feature points in the rectified image are obtained.

[0182] 2. AffNet: Input each stable feature point extracted by KeyNet and the neighborhood image blocks of each stable feature point into AffNet. AffNet can learn affine transformation from the neighborhood image blocks of each stable feature point through convolutional neural network to obtain the mapping transformation relationship between the feature point and the neighborhood image block of the feature point; the mapping transformation relationship describes the local shape of each feature point. By applying the mapping transformation relationship, the normalized neighborhood image block of each feature point in the corrected image can be obtained. Based on the obtained neighborhood image blocks of each feature point in the corrected image, the robustness of the feature description of the corrected image can be improved.

[0183] 3. HardNet: The neighborhood image blocks of each feature point in the corrected image can be input into HardNet. HardNet extracts features from the image blocks through a convolutional neural network and generates a 128-dimensional descriptor, that is, a high-dimensional feature description vector of each feature point in the corrected image is obtained.

[0184] Specifically, the image stitching method provided in this embodiment also includes the process of feature matching and screening:

[0185] AdaLAM (Adaptive Locally Affine Matching) is a feature matching algorithm that can improve the accuracy and robustness of feature matching through adaptive local affine transformation. This algorithm mainly targets local geometric deformations in images and uses local affine models to correct and match feature points. For KeyNet, AffNet, and HardNet, high-dimensional feature descriptors are extracted from images, and AdaLAM is used for feature matching to find the precise corresponding points between images. The feature matching diagram is shown in the figure. Fig. 9 As shown, the pixels in the left image are connected to the corresponding pixels in the right image by line segments, and only six pixels in the left image and the matching pixels in the right image are marked in the figure.

[0186] In addition, RANSAC can be used to eliminate mismatched points. RANSAC randomly selects sample sets and estimates model parameters, iteratively eliminates points that do not conform to the model, thereby finding the optimal set of matching points and ensuring the accuracy of the transformation.

[0187] In an example, the terminal device may determine the quality evaluation result of the stitched image according to the aspect ratio of the stitched image, the detection confidence rate of the stitched image, and the image similarity between each image to be stitched and the matching area of ​​the stitched image.

[0188] In one example, the terminal device first performs a ratio detection, and can determine a reference image in the image sequence, for example, the image to be spliced ​​that is located first in the arrangement order in the image sequence can be determined as the reference image, or the image to be spliced ​​that is located in the middle of the arrangement order can be determined as the reference image; the terminal device can obtain the aspect ratio of the reference image and the aspect ratio of the spliced ​​image, and if the aspect ratio of the spliced ​​image does not change compared with the aspect ratio of the reference image, the terminal device can determine that the quality evaluation result corresponding to the spliced ​​image is a splicing failure result. If the aspect ratio of the spliced ​​image does not change compared with the aspect ratio of the reference image, the terminal device can determine that the spliced ​​image passes the ratio detection.

[0189] The terminal device can perform model training to obtain a target object detection model, namely, a server cabinet detection model, and the terminal device can perform target detection through the server cabinet detection model; if the terminal device determines that the spliced ​​image passes the ratio detection, the terminal device can perform cabinet target detection on the spliced ​​image through the server cabinet detection model, and the result output by the server cabinet detection model can be the detection confidence rate of the spliced ​​image; if the terminal device determines that the currently output detection confidence rate is less than a preset confidence rate threshold, the quality evaluation result corresponding to the spliced ​​image of the terminal device is a splicing failure result. If the currently output detection confidence rate is greater than or equal to the preset confidence rate threshold, the terminal device can determine that the spliced ​​image passes the target detection.

[0190] If it is determined that the stitched image passes the target detection, the terminal device can detect the feature points in the stitched image through the preset feature extraction network in the above embodiment, and determine the matching areas of the stitched image with all the images to be stitched except the reference image through AdaLAM feature matching. The structural similarity (Structural Similarity Index, SSIM) is used to compare the similarity between the image to be stitched and the matching area of ​​the stitched image. If the similarity (i.e., the structural similarity index) between the matching area of ​​the image to be stitched and the stitched image except the reference image is greater than the preset similarity threshold, the quality evaluation result of the stitched image is determined to be a successful stitching result. The preset similarity threshold and the preset confidence rate threshold can be determined based on the actual application scenario. The specific values ​​are not specifically limited in this disclosure.

[0191] like Fig.10 As shown, it can be a flowchart of the implementation process of image stitching and stitching quality assessment based on adaptive feature matching:

[0192] First, three images are taken of the upper, middle and lower sections of the cabinet. The three images taken can be as follows Fig.11 As shown in the figure, the three images are rotated and perspective corrected based on semantic segmentation. After preprocessing, the feature points of the images to be stitched are extracted by combining KeyNet, AffNet and HardNet, and the middle image is used as the reference image to obtain the feature matching pairs of the upper image and the middle image, and the middle image and the lower image for image registration. Finally, the upper image and the lower image are spliced ​​onto the middle image by the fusion method. After the stitching is completed, the stitching quality is tested in turn by the image aspect ratio, confidence and image similarity. The resulting stitched image can be as follows: Fig.11 shown.

[0193] The feature point detection in the image stitching method provided in this embodiment innovatively adopts a combination of KeyNet, AffNet and HardNet to detect image stitching feature points. KeyNet provides stable and consistent key point detection, and can maintain a high key point repetition rate even when the image is rotated, the scale changes, and the illumination changes. AffNet is invariant to changes in viewing angle and perspective deformation. On this basis, HardNet generates a highly discriminative feature descriptor to improve matching accuracy.

[0194] The image preprocessing in the image stitching method provided in this embodiment also uses image rotation correction and perspective correction as preprocessing before stitching, which effectively improves the accuracy of image registration. The stitching quality assessment method in the image stitching method provided in this embodiment is to evaluate the degree of stitching distortion by extracting the feature points of the stitching result image and matching them with the feature points of the image to be stitched, and calculating the structural similarity of the matching area between the image to be stitched and the result image. At the same time, the quality assessment is achieved by comprehensively considering the image aspect ratio and the target detection confidence rate. The introduction of this method effectively reduces manual intervention and realizes the automated image stitching assessment in the computer room scene. In the image stitching method provided in this embodiment, an 82% stitching success rate and an average stitching time of 15.7s are achieved on the computer room image data set by improving the detection and matching of image stitching feature points, which improves the success rate and shortens the stitching time. The image stitching method provided in this embodiment can be used as a pre-processing of all image processing technologies such as image recognition and target detection in the digital twin computer room. The high-quality stitched images generated by this embodiment can significantly improve the accuracy and reliability of subsequent processing.

[0195] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0196] Based on the same inventive concept, the embodiment of the present application also provides an image stitching device for implementing the above-mentioned image stitching method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations of one or more image stitching device embodiments provided below can refer to the above-mentioned limitations on the image stitching method, and will not be repeated here.

[0197] In an exemplary embodiment, Fig.12 As shown, an image stitching device 1200 is provided, comprising:

[0198] A first acquisition module 1202 is used to acquire multiple images to be stitched;

[0199] The first feature processing module 1204 is used to process each image to be spliced ​​through a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to a feature point in each image to be spliced;

[0200] The second feature processing module 1206 is used to perform feature point matching based on the high-dimensional feature description vectors of the feature points in each image to be stitched, so as to obtain similar feature points that meet a preset similarity condition;

[0201] The first stitching module 1208 is used to stitch the images to be stitched based on the similar feature points to obtain a stitched image.

[0202] In one embodiment, the first acquisition module is specifically used for:

[0203] Capturing images of the target object from different shooting angles to obtain a plurality of first images;

[0204] For each of the first images, detect the first image by using a semantic segmentation detection model to obtain mask data of the target object in the first image;

[0205] In the mask data, first pixel points that meet a preset proximity condition with image vertices of the first image are screened out, and contour information of the target object is obtained based on each of the first pixel points;

[0206] The mean value of the angle between the contour information and the horizontal direction is calculated, and the first image is rotated according to the mean value of the angle to obtain an image to be stitched corresponding to the first image.

[0207] In one embodiment, the first acquisition module is specifically used for:

[0208] Rotate the first image by an angle corresponding to the mean of the included angles to obtain a rotationally corrected first image;

[0209] Calculating first perspective transformation parameters based on first position information of each of the first pixels in the first image and second position information of each of the first pixels in the rotationally corrected first image;

[0210] Pixel points other than the first pixel point in the first image are mapped using the first perspective projection transformation parameter to obtain an image to be stitched corresponding to the first image.

[0211] In one embodiment, the first feature extraction module is specifically used for:

[0212] For each of the images to be stitched, image feature learning is performed on the image to be stitched through a first feature extraction network to obtain a feature response map, and non-maximum suppression processing is performed on the feature response map to obtain at least one stable feature point;

[0213] Inputting each of the stable feature points and the neighborhood image blocks of each of the stable feature points into a second feature extraction network, performing affine transformation learning through the second feature extraction network, and obtaining a mapping relationship between the feature points and the neighborhood image blocks; based on the mapping relationship, obtaining the neighborhood image blocks corresponding to each of the feature points in the feature response map;

[0214] The neighborhood image blocks corresponding to each of the feature points in the feature response graph are input into a third feature extraction network to obtain high-dimensional feature description vectors corresponding to each of the feature points.

[0215] In one embodiment, the second feature extraction module is specifically used for:

[0216] For each feature point to be matched in each of the images to be stitched, based on the high-dimensional feature description vector of the feature point to be matched, a feature distance between the high-dimensional feature description vectors of other images to be stitched is calculated to obtain a feature point whose feature distance satisfies a preset similarity condition;

[0217] The feature points in the other images to be stitched are determined as similar feature points in the other images to be stitched to the feature points to be matched in the images to be stitched.

[0218] In one embodiment, the plurality of images to be stitched include a first image to be stitched and a second image to be stitched; each of the similar feature points is a similar feature point of each first feature point on the first image to be stitched on the second image to be stitched; and the first stitching module is specifically configured to:

[0219] Based on the position information of the feature point in the first image to be stitched, and the position information of the feature point similar to the feature point in the second image to be stitched, obtaining a second perspective transformation parameter;

[0220] Performing perspective transformation on each second feature point on the first image to be stitched based on the second perspective transformation parameter to obtain a transformed first image to be stitched, wherein the transformed first image to be stitched has the same position coordinate system as the second image to be stitched;

[0221] The overlapped pixels in the transformed first image to be stitched and the second image to be stitched are fused to obtain a stitched image of the first image to be stitched and the second image to be stitched.

[0222] In an exemplary embodiment, Fig.13 As shown, an image stitching device 1300 is provided, comprising:

[0223] The second acquisition module 1302 is used to acquire an image sequence corresponding to the target object, where the image sequence includes a plurality of images to be stitched arranged in order of size of image acquisition angles;

[0224] A third acquisition module 1304 is used to acquire adjacent first images to be stitched and second images to be stitched in the image sequence, and process the adjacent first images to be stitched and second images to be stitched according to any method of the first aspect to obtain a stitching result of the first images to be stitched and the second images to be stitched;

[0225] The updating module 1306 is used to update the image sequence based on the stitching result to obtain an updated image sequence, and re-execute the step of obtaining adjacent first images to be stitched and second images to be stitched in the image sequence based on the updated image sequence until a stitched image of the image sequence is obtained.

[0226] In one embodiment, the update module is specifically used to:

[0227] The first image to be stitched and the second image to be stitched are replaced by the stitching result to obtain an updated image sequence.

[0228] In one embodiment, the device further comprises:

[0229] The evaluation module is used to obtain a quality evaluation result of the stitched image through one or more of the aspect ratio of the stitched image, the detection confidence rate of the stitched image, and the image similarity between each of the images to be stitched and the matching area of ​​the stitched image.

[0230] Each module in the above-mentioned image stitching device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0231] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.14As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image stitching method is implemented.

[0232] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0233] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps described in the above embodiment when executing the computer program.

[0234] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above embodiment are implemented.

[0235] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above embodiments when executed by a processor.

[0236] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0237] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processing unit, a data processing logic based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited thereto.

[0238] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0239] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image stitching method, characterized in that: The method comprises: Obtain multiple images to be stitched; Processing each of the images to be spliced ​​by a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to a feature point in each of the images to be spliced; Based on the high-dimensional feature description vectors of the feature points in each of the images to be stitched, feature point matching is performed to obtain similar feature points that meet preset similarity conditions; The images to be stitched are stitched based on the similar feature points to obtain a stitched image.

2. The method according to claim 1, characterized in that: The step of obtaining a plurality of images to be stitched comprises: For each of the multiple first images, detect the first image by using a semantic segmentation detection model to obtain mask data of the target object in the first image; In the mask data, first pixel points that meet a preset proximity condition with image vertices of the first image are screened out, and contour information of the target object is obtained based on each of the first pixel points; The mean value of the angle between the contour information and the horizontal direction is calculated, and the first image is rotated according to the mean value of the angle to obtain an image to be stitched corresponding to the first image.

3. The method according to claim 2, characterized in that The step of rotating the first image by using the angle mean to obtain an image to be stitched corresponding to the first image includes: Rotate the first image by an angle corresponding to the mean of the included angles to obtain a rotationally corrected first image; Calculating first perspective transformation parameters based on first position information of each of the first pixels in the first image and second position information of each of the first pixels in the rotationally corrected first image; Pixel points other than the first pixel point in the first image are mapped using the first perspective projection transformation parameter to obtain an image to be stitched corresponding to the first image.

4. The method according to claim 1, characterized in that The processing of each of the images to be spliced ​​by a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to a feature point in each of the images to be spliced ​​includes: For each of the images to be stitched, image feature learning is performed on the image to be stitched through a first feature extraction network to obtain a feature response map, and non-maximum suppression processing is performed on the feature response map to obtain at least one stable feature point; Inputting each of the stable feature points and the neighborhood image blocks of each of the stable feature points into a second feature extraction network, performing affine transformation learning through the second feature extraction network, and obtaining a mapping relationship between the feature points and the neighborhood image blocks; based on the mapping relationship, obtaining the neighborhood image blocks corresponding to each of the feature points in the feature response map; The neighborhood image blocks corresponding to each of the feature points in the feature response graph are input into a third feature extraction network to obtain high-dimensional feature description vectors corresponding to each of the feature points.

5. The method according to claim 1, characterized in that The high-dimensional feature description vector based on the feature points in each of the images to be stitched is used to match the feature points to obtain similar feature points that meet the preset similarity conditions, including: For each feature point to be matched in each of the images to be stitched, based on the high-dimensional feature description vector of the feature point to be matched, a feature distance between the high-dimensional feature description vectors of other images to be stitched is calculated to obtain a feature point whose feature distance satisfies a preset similarity condition; The feature points in the other images to be stitched are determined as similar feature points in the other images to be stitched to the feature points to be matched in the images to be stitched.

6. The method according to claim 2, characterized in that The plurality of images to be stitched include a first image to be stitched and a second image to be stitched; each of the similar feature points is a similar feature point of each first feature point on the first image to be stitched on the second image to be stitched; The step of stitching the images to be stitched based on the similar feature points to obtain a stitched image includes: Obtaining a second perspective transformation parameter based on position information of a feature point in the first image to be stitched and position information of a feature point similar to the feature point in the second image to be stitched; Performing perspective transformation on each second feature point on the first image to be stitched based on the second perspective transformation parameter to obtain a transformed first image to be stitched, wherein the transformed first image to be stitched has the same position coordinate system as the second image to be stitched; The overlapped pixels in the transformed first image to be stitched and the second image to be stitched are fused to obtain a stitched image of the first image to be stitched and the second image to be stitched.

7. An image stitching method, characterized in that: The method comprises: Acquire an image sequence corresponding to the target object, wherein the image sequence includes a plurality of images to be stitched arranged in order of size of image acquisition angles; In the image sequence, an adjacent first image to be stitched and a second image to be stitched are obtained, and according to the method according to any one of claims 1 to 6, the adjacent first image to be stitched and the second image to be stitched are processed to obtain a stitching result of the first image to be stitched and the second image to be stitched; The image sequence is updated based on the stitching result to obtain an updated image sequence, and based on the updated image sequence, the step of acquiring adjacent first images to be stitched and second images to be stitched in the image sequence is re-executed until a stitched image of the image sequence is obtained.

8. The method according to claim 7, characterized in that The updating of the image sequence based on the stitching result to obtain an updated image sequence includes: The first image to be stitched and the second image to be stitched are replaced by the stitching result to obtain an updated image sequence.

9. The method according to claim 7, characterized in that: The method further comprises: The quality evaluation result of the stitched image is obtained through one or more of the aspect ratio of the stitched image, the detection confidence rate of the stitched image, and the image similarity between each of the images to be stitched and the matching area of ​​the stitched image.

10. An image stitching device, characterized in that: The device comprises: A first acquisition module, used for acquiring a plurality of images to be stitched; A first feature processing module is used to process each of the images to be stitched through a preset feature extraction network to obtain a high-dimensional feature description vector corresponding to a feature point in each of the images to be stitched; A second feature processing module is used to perform feature point matching based on the high-dimensional feature description vectors of the feature points in each of the images to be stitched, so as to obtain similar feature points that meet a preset similarity condition; The first stitching module is used to stitch the images to be stitched based on the similar feature points to obtain a stitched image.

11. An image stitching device, characterized in that: The device comprises: A second acquisition module is used to acquire an image sequence corresponding to the target object, wherein the image sequence includes a plurality of images to be stitched arranged in order of size of image acquisition angles; a third acquisition module, configured to acquire, in the image sequence, an adjacent first image to be stitched and a second image to be stitched, and process the adjacent first image to be stitched and the second image to be stitched according to the method according to any one of claims 1 to 6 to obtain a stitching result of the first image to be stitched and the second image to be stitched; An updating module is used to update the image sequence based on the stitching result to obtain an updated image sequence, and based on the updated image sequence, re-execute the step of acquiring adjacent first images to be stitched and second images to be stitched in the image sequence until a stitched image of the image sequence is obtained.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images