Image splicing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image splicing method and device, electronic equipment and a readable storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a first image and a second image obtained through the two times of screen capture of the electronic equipment in response to the screen capture operation of a user on the electronic equipment; performing down-sampling on the first image and the second image to obtain a third image corresponding to the first image and a fourth image corresponding to the second image; extracting feature points, meeting a preset feature point condition, of the third image and the fourth image to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image; based on the first feature point set and the second feature point set, at least one matching feature pair is determined, and each matching feature pair comprises a first target feature point and a second target feature point which are matched with each other; and based on the at least one matching feature pair, splicing the first image and the second image to obtain a target image. And the image splicing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image stitching method, device, electronic device and readable storage medium. Background Art

[0002] At present, with the development of electronic information technology, users can stitch images. However, the current image stitching methods have high computational complexity and are time-consuming, resulting in low stitching efficiency. Summary of the invention

[0003] The present application proposes an image stitching method, device, electronic device and readable storage medium.

[0004] In a first aspect, an embodiment of the present application provides an image stitching method, including: in response to a user's screenshot operation on an electronic device, acquiring a first image and a second image obtained by two screenshots of the electronic device; downsampling the first image and the second image to obtain a third image corresponding to the first image and a fourth image corresponding to the second image; extracting feature points of the third image and the fourth image that meet preset feature point conditions to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image; based on the first feature point set and the second feature point set, determining at least one matching feature pair, wherein each matching feature pair includes a first target feature point and a second target feature point that match each other; based on at least one matching feature pair, stitching the first image and the second image to obtain a target image.

[0005] In the second aspect, the embodiment of the present application also provides an image stitching device, including: an acquisition unit, a resizing unit, a feature point extraction unit, a feature matching unit and a stitching unit. The acquisition unit is used to acquire the first image and the second image obtained by two screenshots of the electronic device in response to the user's screenshot operation on the electronic device; the resizing unit is used to downsample the first image and the second image to obtain a third image corresponding to the first image and a fourth image corresponding to the second image; the feature point extraction unit is used to extract feature points of the third image and the fourth image that meet the preset feature point conditions to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image; the feature matching unit is used to determine at least one matching feature pair based on the first feature point set and the second feature point set, wherein each matching feature pair includes a first target feature point and a second target feature point that match each other; the stitching unit is used to stitch the first image and the second image based on at least one matching feature pair to obtain a target image.

[0006] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method described in the first aspect above.

[0008] The image stitching method, device, electronic device and readable storage medium provided in the embodiment of the present application first respond to the screenshot operation of the electronic device by the user, obtain the first image and the second image obtained by two screenshots of the electronic device; downsample the first image and the second image to obtain the third image corresponding to the first image and the fourth image corresponding to the second image; then extract the feature points of the third image and the fourth image that meet the preset feature point conditions to obtain the first feature point set corresponding to the third image and the second feature point set corresponding to the fourth image; then determine at least one matching feature pair based on the first feature point set and the second feature point set, wherein each matching feature pair includes a first target feature point and a second target feature point that match each other; thereby, based on at least one matching feature pair, the first image and the second image are stitched to obtain the target image. First, in the scheme of the present application, the objects for feature point extraction are the downsampled third image and the fourth image, then compared with the first image and the second image, the third image and the fourth image have a smaller size, thereby saving processing time and improving the overall efficiency of image stitching. In addition, after obtaining the target feature pair, the third image and the fourth image after downsampling are not directly used for stitching, but the first image and the second image before downsampling are returned for stitching, so that the stitching effect can also be guaranteed.

[0009] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 An application scenario diagram of the image stitching method provided in an embodiment of the present application is shown;

[0012] Figure 2 A method flow chart of an image stitching method provided by an embodiment of the present application is shown;

[0013] Figure 3 A method flow chart of an image stitching method provided by another embodiment of the present application is shown;

[0014] Figure 4 A schematic diagram showing the implementation of the present application to determine the cropping area;

[0015] Figure 5 A method flow chart of an image stitching method provided by another embodiment of the present application is shown;

[0016] Figure 6 A schematic diagram showing the comparison of the results of the image stitching method provided by the embodiment of the present application and the current image stitching method is shown;

[0017] Figure 7 A schematic diagram showing a comparison of results of an image stitching method provided by another embodiment of the present application and a current image stitching method;

[0018] Figure 8 A schematic diagram showing a comparison of results of an image stitching method provided by another embodiment of the present application and a current image stitching method;

[0019] Fig. 9 A schematic diagram showing a comparison of the results of an image stitching method provided by yet another embodiment of the present application and a current image stitching method is shown;

[0020] Fig.10 The structure block diagram of the image stitching device provided by the embodiment of the present application is shown;

[0021] Fig.11 A structural block diagram of an electronic device provided in an embodiment of the present application is shown;

[0022] Fig.12 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0024] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0025] At present, with the development of electronic information technology, users can stitch images. However, the current image stitching methods have high computational complexity and long time consumption, resulting in low stitching efficiency. How to reduce the complexity of image stitching, improve the efficiency of image stitching and reduce the time consumption is an urgent problem to be solved.

[0026] At present, images can be stitched based on deep learning image stitching algorithms. Specifically, deep learning models can be used to extract, match and fuse features, which is suitable for processing complex image stitching scenarios. Among them, deep learning models can be models such as Convolutional Neural Network (CNN), Generative Adversarial Network (GAN), Recurrent Neural Network (RNN), etc.

[0027] However, the inventors found in their research that although the stitching algorithm based on deep learning has a certain degree of accuracy, it has high computational complexity and poor real-time performance, and it is difficult to achieve real-time stitching in resource-constrained environments such as mobile terminals.

[0028] Therefore, in order to solve or partially solve the above problems, the present application provides an image stitching method, device, electronic device and readable storage medium.

[0029] See also Figure 1 , Figure 1An application scenario diagram of the image stitching method provided in an embodiment of the present application is shown, namely, an image stitching scenario 100 , which includes an electronic device 110 and a user 120 .

[0030] exist Figure 1 The electronic device 110 shown in the figure is a smart phone, and the user 120 can use the electronic device 110. The electronic device 110 can run an operating system, so that the user 120 can perform the image stitching method by setting it on the electronic device 110. Exemplarily, the audio processing method may include performing a screenshot operation on the electronic device, obtaining a first image and a second image, and then stitching the first image and the second image by the electronic device. For an introduction to the image stitching method, please refer to the subsequent embodiments.

[0031] See also Figure 2 , Figure 2 A method flow chart of an image stitching method provided in an embodiment of the present application is shown. The image stitching method can be applied to Figure 1 The electronic device in the image stitching scene 100 shown in the figure may specifically use a processor of the electronic device as an execution subject for executing the image stitching method. The image stitching method may include steps S110 to S150.

[0032] Step S110: In response to a user's screenshot operation on the electronic device, a first image and a second image obtained by two screenshots of the electronic device are acquired.

[0033] The image stitching method provided in the embodiment of the present application can be applied to the scene of taking screenshots of electronic devices, such as the scene of long screenshots. In the scene of long screenshots, an image is generated by taking screenshots of the display screen of the electronic device, which generally cannot fully display the content, so it is necessary to control the screen scrolling of the electronic device and continuously take screenshots of the scrolling screen to obtain multiple images, and then stitch the obtained multiple images to obtain the image required in the long screenshot scene.

[0034] In some embodiments, in response to a user's screen capture operation on an electronic device, a screen capture scene may be entered, and a screen capture image may be obtained, wherein the screen capture image may be generated based on the content displayed on the screen of the electronic device when the screen capture operation is triggered. For example, the content displayed on the screen of the electronic device when the screen capture operation is triggered may be directly used as the content of the screen capture image; for another example, the content displayed on the screen of the electronic device when the screen capture operation is triggered may be image processed to obtain the content as the screen capture image. The image processing may include image marking, image graffiti, etc.

[0035] The user may trigger the screenshot operation by operating the target control. For example, the target control may be a virtual control displayed on the screen. Specifically, when the electronic device detects that it is in a screenshot scene, a virtual control may be displayed at a specified position on the screen, and the user may input the screenshot operation by interacting with the virtual control. In another example, the target control may also be a physical button. Specifically, it may be a physical button or a combination of multiple physical buttons. The user may input the screenshot operation by pressing a physical button or multiple physical buttons.

[0036] For some embodiments, by inputting a screenshot operation once, the electronic device can be controlled to perform two screenshots, thereby obtaining a first image and a second image. Alternatively, by inputting two screenshot operations, the electronic device responds to one screenshot operation each time, and obtains an image corresponding to one screenshot operation. In other words, the first image and the second image can be obtained by inputting two screenshot operations.

[0037] For example, when a scrolling operation is input to the screen of the electronic device, the user may trigger two consecutive screenshot operations on the electronic device, and the electronic device thereby obtains a first image and a second image. The scrolling operation may be used to trigger the screen of the electronic device to scroll. It is understandable that the screen of the electronic device may scroll in different directions, for example, the electronic device may have a first side and a second side that are relatively arranged, the first side may be the upper side of the electronic device, and the second side may be the lower side of the electronic device. Thus, the first direction in which the first side points to the second side may be used as the direction of the scrolling operation.

[0038] In addition, the electronic device may also automatically trigger a screenshot operation. Therefore, in some embodiments, the electronic device may automatically trigger a scrolling operation to generate a screen, so that it can be detected that the screen of the electronic device has been input with a scrolling operation. Then, in the case where the screen of the electronic device has been input with a scrolling operation, the screenshot operation is further automatically triggered, for example, two consecutive screenshot operations are triggered, so that a first image and a second image can be obtained. It can be understood that the first image is a screenshot image generated by a previous screenshot operation; and the second image is a screenshot image generated by a subsequent screenshot operation. The electronic device thereby obtains the first image and the second image generated by two consecutive screenshot operations.

[0039] The scrolling operation may be automatically triggered when the electronic device detects that the screen is currently in a screenshot scene, so that the electronic device can detect that a scrolling operation has been input to the screen. The scrolling operation may also be input by a user, for example, the user slides a finger on the screen to input a scrolling operation to the screen of the electronic device.

[0040] Step S120: down-sample the first image and the second image to obtain a third image corresponding to the first image and a fourth image corresponding to the second image.

[0041] After the first image and the second image are acquired, if the first image and the second image are directly used for subsequent image matching, it may take a long time and reduce the image matching efficiency.

[0042] Therefore, in some implementations, the image sizes of the first image and the second image may be reduced respectively to obtain a third image corresponding to the first image and a fourth image corresponding to the second image.

[0043] Specifically, the first image and the second image may be downsampled respectively to obtain a third image corresponding to the first image and a fourth image corresponding to the second image, so as to reduce the sizes of the first image and the second image.

[0044] Optionally, after cropping the first image and the second image, the cropped first image and the second image may be downsampled respectively, thereby further reducing the size of the first image and the second image. Detailed description can be found in the subsequent embodiments.

[0045] Step S130: extracting feature points of the third image and the fourth image that meet preset feature point conditions to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image.

[0046] It is understandable that extracting feature points from the third image and the fourth image is essentially screening each pixel in the third image and the fourth image to find pixels that meet the preset feature point conditions as feature points. Exemplarily, the preset feature point conditions may include that the grayscale change of the pixel point is obvious, the repeatability is high, and the corresponding feature vector has a distinguishing degree, etc.

[0047] It can be understood that the first feature point set corresponding to the third image substantially includes each pixel point determined as a feature point in the third image. Optionally, the first feature point set may also include image features of each feature point. The image features may be represented in the form of a feature vector with a channel number of C, a width of W, and a height of H.

[0048] That is, the first feature point set includes specific feature points of the third image and image features corresponding to each feature point. Similarly, the second feature point set includes specific feature points of the fourth image and image features corresponding to each feature point.

[0049] For some implementations, feature points that meet preset feature point conditions of the third image and the fourth image may be extracted by a feature extraction model to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image. For a detailed description, please refer to the subsequent embodiments.

[0050] Optionally, since the structure and parameters of the feature extraction model required for extracting feature points from the third image and the fourth image are the same, a twin network with shared parameters can be constructed to extract feature points from the third image and the fourth image to improve computational efficiency and reduce storage space requirements. For a detailed description, please refer to the subsequent embodiments.

[0051] Step S140: Determine at least one matching feature pair based on the first feature point set and the second feature point set, wherein each matching feature pair includes a first target feature point and a second target feature point that match each other.

[0052] It can be understood that the first feature point set includes multiple first feature points, and the first feature points are the feature points obtained by extracting feature points from the first image; similarly, the second feature point set includes multiple second feature points, and the second feature points are the feature points obtained by extracting feature points from the second image.

[0053] Furthermore, at least one matching feature pair may be determined based on the first feature point set and the second feature point set. Specifically, feature point matching may be performed on each first feature point in the first feature point set and each second feature point in the second feature point set to find mutually matching first target feature points and second target feature points. Mutually matching first target feature points and second target feature points are the matching feature pair.

[0054] In some implementations, the first feature point set and the second feature point set may be first screened to screen out some of the first feature points as first target feature points, and screen out some of the second feature points as second target feature points. Then, in order to find mutually matching first target feature points and second target feature points, the similarity between each first target feature point and each second target feature point may be determined, and then the mutually matching first target feature points and second target feature points may be determined based on the similarity. For a detailed description, please refer to the subsequent embodiments.

[0055] It should be noted that the matching feature pair may be one or more.

[0056] Step S150: Based on at least one of the matching feature pairs, the first image and the second image are spliced ​​to obtain a target image.

[0057] After obtaining at least one matching feature pair based on the above steps, if the third image and the fourth image are spliced ​​directly based on the feature matching pair, since the third image and the fourth image are images with reduced image sizes, the obtained spliced ​​image may have poor effect, such as low clarity or partial image loss.

[0058] Therefore, for some implementations, a first matching region can be determined in the first image and a second matching region can be determined in the second image based on the matching feature pairs determined in the third image and the fourth image, respectively. Then, the first image and the second image are spliced ​​based on the first matching region and the second matching region to obtain a target image. For a detailed description, please refer to the subsequent embodiments.

[0059] The image stitching method provided in the embodiment of the present application first responds to the screenshot operation of the electronic device by the user, obtains the first image and the second image obtained by two screenshots of the electronic device; downsamples the first image and the second image to obtain the third image corresponding to the first image and the fourth image corresponding to the second image; then extracts the feature points of the third image and the fourth image that meet the preset feature point conditions to obtain the first feature point set corresponding to the third image and the second feature point set corresponding to the fourth image; then based on the first feature point set and the second feature point set, at least one matching feature pair is determined, wherein each matching feature pair includes a first target feature point and a second target feature point that match each other; thereby, based on at least one matching feature pair, the first image and the second image are stitched to obtain the target image. First, in the scheme of the present application, the objects for feature point extraction are the downsampled third image and the fourth image, then compared with the first image and the second image, the third image and the fourth image have a smaller size, thereby saving processing time and improving the overall efficiency of image stitching. In addition, after obtaining the target feature pair, the third image and the fourth image after downsampling are not directly used for stitching, but the first image and the second image before downsampling are returned for stitching, so that the stitching effect can also be guaranteed.

[0060] See also Figure 3 , Figure 3 A method flow chart of an image stitching method provided in an embodiment of the present application is shown. The image stitching method can be applied to Figure 1 The electronic device in the image stitching scene 100 shown in the figure may specifically use a processor of the electronic device as an execution subject for executing the image stitching method. The image stitching method may include steps S210 to S2100.

[0061] Step S210: In response to a user's screenshot operation on the electronic device, a first image and a second image obtained by two screenshots of the electronic device are acquired.

[0062] Among them, step S210 has been introduced in detail in the above embodiment and will not be repeated here.

[0063] Step S220: determining a first cropping region in the first image, and determining a second cropping region in the second image.

[0064] A first cropping area may be determined in the first image, and a second cropping area may be determined in the second image. Thus, an image other than the first cropping area in the first image may be used as the third image, and an image other than the second cropping area in the second image may be used as the fourth image, so as to reduce the image sizes of the first image and the second image.

[0065] It is understandable that the electronic device includes a first side and a second side that are opposite to each other. The direction of the scrolling operation can be set to a first direction from the first side to the second side, and since the first image and the second image are obtained by two consecutive screenshot operations of the electronic device, the screen display content corresponding to the first image and the screen display content corresponding to the second image can be distributed in sequence along the first direction.

[0066] For example, see Figure 4 , Figure 4 A schematic diagram of determining a cropping area according to the present application is shown. Figure 4 The electronic device 110 shown in FIG. 1 includes a first side 111 and a second side 112 opposite to each other. Therefore, it can be determined that a first direction 1101 in which the side 111 points to the second side 112 is defined.

[0067] Therefore, step S220 may also include steps S221 to S224.

[0068] Step S221: determining a first sideline corresponding to the first side in the first image.

[0069] Step S222: determining a first cropping area in the first image along the first direction based on the first edge line.

[0070] Step S223: determining a second sideline corresponding to the second side in the second image.

[0071] Step S224: determining a second cropping area in the second image based on the second edge line in a direction opposite to the first direction.

[0072] Please continue reading Figure 4 ,exist Figure 4 The first image 410 and the second image 420 are also shown. In some embodiments, the direction of the scrolling operation of the display screen of the electronic device may be a first direction. Thus, the screen display content corresponding to the first image 410 and the screen display content corresponding to the second image 420 are sequentially distributed along the first direction.

[0073] A first sideline 419 corresponding to the first side edge 111 may be determined in the first image 410. Then, a first cropping area 412 is determined in the first image 410 along the first direction 1101 based on the first sideline 419.

[0074] Exemplarily, the area of ​​the first image 410 swept by the first sideline 419 moving along the first direction 1101 for the first distance can be used as the first cropping area 412. That is, the area between the first sideline 419 and the dotted line 411 in the first image 410. It should be noted that the dotted line 411 is only used to illustrate the first cropping area 412, and in actual application, the dotted line 411 is invisible to the user.

[0075] Similarly, a second sideline 429 corresponding to the second side edge 112 may be determined in the second image 420 . Then, a second cropping area 422 is determined in the second image 420 based on the second sideline 429 in the opposite direction of the first direction 1101 .

[0076] Exemplarily, the area of ​​the second image 420 swept by the second sideline 429 moving the second distance in the opposite direction of the first direction 1101 can be used as the second cropping area 422. That is, the area between the second sideline 429 and the dotted line 421 in the second image 420. It should be noted that the dotted line 421 is only used to illustrate the second cropping area 422, and in actual application, the dotted line 421 is invisible to the user.

[0077] Optionally, the distance between the first side 111 and the second side 112 may be set as the third distance. Thus, the first distance is set to be less than or equal to 1 / N1 of the third distance, and the second distance is set to be less than or equal to 1 / N2 of the third distance. Wherein, N1 and N2 are integers greater than 4 respectively.

[0078] Step S230: down-sampling the image except the first cropping area in the first image to obtain the third image, and down-sampling the image except the second cropping area in the second image to obtain the fourth image.

[0079] After determining the first cropping area and the second cropping area through the above steps, the image of the first image excluding the first cropping area can be used as the third image, and the image of the second image excluding the second cropping area can be used as the fourth image.

[0080] Please continue reading Figure 4 In the first image 410, the area 413 except the first cropping area 412 can be used as the third image. In the second image 420, the area 423 except the second cropping area 422 can be used as the fourth image.

[0081] Optionally, in order to further reduce the size of the image subsequently input into the feature extraction model and improve the overall rate of image stitching, the image outside the first cropped area in the first image may be downsampled to obtain a third image; and the image outside the second cropped area in the second image may be downsampled to obtain a fourth image.

[0082] Exemplarily, the length and width of the image in the first image excluding the first cropping area may be reduced to 1 / M. Similarly, the length and width of the image in the second image excluding the second cropping area may be reduced to 1 / M, for example, M may be 2, 3, etc.

[0083] It should be noted that downsampling essentially means reducing the number of pixels in an image.

[0084] Optionally, in order to ensure the effect of subsequent feature point extraction through the feature extraction model, in some embodiments, the value of the aforementioned M can also be set so that the number of pixels in length and width of the downsampled third image and the fourth image is an integer multiple of 32.

[0085] In the image stitching method provided in the embodiment of the present application, invalid feature points are removed from the obtained third image and the fourth image, which can reduce interference with stitching accuracy and improve the efficiency of image stitching.

[0086] Step S240: extracting feature points of the third image that meet preset feature point conditions through the first twin network branch in the feature extraction model to obtain a first feature point set corresponding to the third image.

[0087] Step S250: extracting feature points of the fourth image that meet preset feature point conditions through the second twin network branch in the feature extraction model to obtain a second feature point set corresponding to the fourth image, wherein the first twin network branch and the second twin network branch share parameters.

[0088] From the foregoing introduction, it can be known that the feature extraction model can be used to extract feature points of the third image and the fourth image that meet preset feature point conditions to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image.

[0089] The feature extraction model may be a pre-trained model. In some embodiments, the trained model may be an improved XFeat model. Exemplarily, the XFeat model may be light-weighted and a depth-separable convolution may be used to efficiently extract image features.

[0090] In some embodiments, a training data set for training a model may be obtained in advance. For example, the training data set may be a plurality of pairs of first training images and second training images obtained by obtaining two consecutive screenshot operations of the electronic device in screenshot scenarios when the screen of the electronic device is scrolled. Then, the standard matching areas in each pair of the first training image and the second training image are marked respectively, and the standard feature points that match each other in the first training image and the second training image are marked, and the standard feature points that match each other in each pair of the first training image and the second training image can constitute a standard matching feature pair.

[0091] Therefore, after obtaining the training data set, the model can be trained using the training data set. For example, a pair of first training images and second training images can be used as the input of the model each time to obtain the initial matching area, initial matching points, and initial matching feature pairs of the first training image and the second training image output by the model. Then, the first difference between the standard matching area and the initial matching area, the second difference between the standard matching point and the initial matching point, and the third difference between the standard matching feature pair and the initial matching feature pair are obtained respectively, so as to train the model based on the first difference, the second difference, and the third difference to reduce the first difference, the second difference, and the third difference. The trained model is the feature extraction model.

[0092] By obtaining a training data set in a screenshot scenario to train the model to obtain a feature extraction model, the accuracy of feature point extraction using the feature extraction model in the subsequent screenshot scenario can be improved to a certain extent.

[0093] Furthermore, in order to improve the efficiency of feature point extraction and reduce the demand for storage space, a twin network with shared parameters can also be constructed to extract feature points of the first image and the second image. In some embodiments, the twin network may include a first twin network branch and a second twin network branch with shared parameters. Thus, the first twin network branch in the feature extraction model can be used to extract feature points of the third image that meet the preset feature point conditions, and obtain a first feature point set corresponding to the third image; the second twin network branch in the feature extraction model can be used to extract feature points of the fourth image that meet the preset feature point conditions, and obtain a second feature point set corresponding to the fourth image.

[0094] The shared parameters may include weights.

[0095] It should be noted that the feature point extraction of the third image through the first twin network branch and the feature point extraction of the fourth image through the second twin network branch can be performed in parallel.

[0096] It is understandable that storage space is required to store the parameters of the model (network branch). Therefore, in the method provided in the embodiment of the present application, by constructing a twin network, the first twin network and the second twin network can share parameters without training or storing two independent sets of parameters, which can reduce the demand for storage space. In addition, the first twin network and the second twin network can perform feature point extraction in parallel, thereby effectively improving the overall efficiency of image stitching.

[0097] In addition, before step S240, a feature extraction model can also be obtained, so that the feature extraction model can be used to extract feature points of the third image and the fourth image that meet preset feature point conditions to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image.

[0098] Therefore, for some implementation modes, before step S240, steps S231 to S233 may also be included.

[0099] S231: Obtain a first model for extracting features from an image.

[0100] S232: Perform terminal-side model transplantation and quantization processing on the first model to obtain a second model.

[0101] S233: Perform parallel acceleration processing on the second model to obtain the feature extraction model.

[0102] It is understandable that for mobile electronic devices, such as smart phones, smart tablets, laptops, etc., the processing power and storage capacity of such electronic devices are limited. Therefore, in order to better adapt to the deployment of feature extraction models in electronic devices, a first model for feature extraction of images can be first obtained.

[0103] In some implementations, the first model may be obtained by training the model using the training data set shown in the aforementioned embodiment. Thus, after obtaining the first model, the first model may be subjected to end-side model transplantation and quantization processing to obtain the second model. The trained model may be an XFeat model improved by lightweight quantization, which has the ability to extract feature points from an image through deep separable convolution.

[0104] For example, the model can be trained under the Pytorch framework to obtain a first model. Then, the first model is transplanted to the NCNN framework for end-side optimization on the mobile terminal. In addition, the first model after the end-side model transplantation can be quantized, and then the quantized first model can be used as the second model. For example, INT8 quantization can be performed.

[0105] In order to further improve the inference rate of the model deployed in the electronic device, the second model can also be processed in parallel to obtain the feature extraction model. Exemplarily, the second model can be processed in parallel by OpenMP (Open Multi-Processing) parallel acceleration technology.

[0106] Optionally, after obtaining the first model as mentioned above, an initial twin network with shared parameters can also be constructed based on the first model. The initial twin network can include the first initial twin network and the second initial twin network, so as to perform end-side model transplantation, quantization processing and parallel acceleration processing on the initial twin network to obtain the first twin network corresponding to the first initial twin network and the second twin network corresponding to the second initial twin network.

[0107] Therefore, the obtained feature extraction model can be deployed in electronic devices, realizing efficient deployment of the feature extraction model in mobile electronic devices.

[0108] Optionally, before deploying the feature extraction model to the electronic device, the feature matching model can also be pruned, distilled, and improved with a lightweight model, thereby further reducing the number of parameters and computational complexity of the feature extraction model, improving the model inference speed, and reducing the model inference time while ensuring that the accuracy of the feature extraction model is almost not lost.

[0109] Step S260: performing screening operations on the first feature point set and the second feature point set respectively to obtain first target feature points and second target feature points that meet target conditions.

[0110] It is understandable that in order to improve the efficiency of determining matching feature pairs and thus improve the overall efficiency of image stitching, the first feature point set and the second feature point set may be screened first, and then matching may be performed based on the screened feature points.

[0111] In some implementations, the feature points may be screened by the confidence score of each feature point. Specifically, step S260 may include steps S261 to S263.

[0112] Step S261: Obtain a first confidence score for each of the first feature points and a second confidence score for each of the second feature points.

[0113] Step S262: Filter out a specified number of first feature points corresponding to the highest first confidence scores as first feature points that meet the target condition.

[0114] Step S263: Filter out a specified number of second feature points corresponding to the second highest confidence scores as second feature points that meet the target condition.

[0115] In addition to determining each first feature point in the first feature point set by using the aforementioned feature extraction model, the reliability of each first feature point may also be determined, for example, by characterizing the reliability of each first feature point by using a reliability score.

[0116] In some embodiments, the reliability score may be a confidence score. For example, when extracting feature points from the third image using a feature extraction model, the feature extraction model may also output a first confidence score corresponding to each first feature point.

[0117] Optionally, when the feature extraction model extracts feature points of the third image, it can also output a heat map corresponding to the feature points in the third image, so that the first confidence score corresponding to each first feature point can be obtained based on the numerical value of each key point in the heat map and the feature vector representing the feature point.

[0118] Similarly, the second confidence score corresponding to the second feature point can also be obtained through the above method.

[0119] Furthermore, a specified number of first feature points corresponding to the highest first confidence scores may be selected from the first feature point set as first target feature points that meet the target condition. For example, the first feature points may be sorted from high to low based on the first confidence scores, and then a specified number of first feature points corresponding to the highest first confidence scores may be selected from high to low as first target feature points that meet the target condition. For example, the specified number may be 256.

[0120] Similarly, a specified number of second target feature points corresponding to the highest second confidence scores may be screened out from the second feature point set. For a detailed description, please refer to the method for screening the first feature point set, which will not be repeated here.

[0121] Optionally, after obtaining the first feature point set, non-maximum suppression (NMS) processing can be performed on each first feature point in the first feature point set to remove redundant first feature points and retain representative and significant first feature points as much as possible. Then, the above screening operation is performed on the first feature point set that has been subjected to the non-maximum suppression processing to obtain the first target feature points that meet the target conditions.

[0122] Similarly, after obtaining the second feature point set, non-maximum suppression processing can be first performed on each second feature point in the second feature point set, and then the above-mentioned screening operation can be performed on the second feature point set that has undergone non-maximum suppression processing to obtain the second target feature points that meet the target conditions.

[0123] Therefore, the first target feature point and the second target feature point can be directly matched subsequently, which reduces the processing time required for matching and improves the matching accuracy to a certain extent, that is, improves the accuracy of the obtained matching feature pairs.

[0124] Step S270: Determine the similarity between each first target feature point and each second target feature point.

[0125] Step S280: Determine the second target feature point with the highest similarity corresponding to each first target feature point as the second target feature point that matches the first target feature point, so as to obtain at least one matching feature pair.

[0126] Furthermore, in order to match each first target feature point and each second target feature point, the similarity between each first target feature point and each second target feature point may be determined.

[0127] For some implementations, the similarity between each first target feature point and each second target feature point can be obtained by calculating the cosine similarity between each first target feature point and each second target feature point. That is, the similarity can be the cosine similarity.

[0128] Therefore, based on the similarity between the first target feature point and each second target feature point, the second target feature point with the highest similarity to the first target feature point can be found as the second target feature point that matches the first target feature point. The first target feature point and the second target feature point that matches the first target feature point can form a matching feature pair.

[0129] Based on the above method, the second target feature points respectively matching each first target feature point can be determined in sequence.

[0130] Step S290: Determine a first matching area in the first image and a second matching area in the second image based on the matching feature pair.

[0131] Further, after obtaining the matching feature pair, a first matching area can be determined in the first image and a second matching area can be determined in the second image based on the matching feature pair.

[0132] It is understandable that, since the third image is obtained by reducing the size of the first image, and the fourth image is obtained by reducing the size of the second image, the first image and the third image may have a first mapping relationship, and the second image and the fourth image may have a second mapping relationship. The first mapping relationship may be used to map the positional relationship in the first image and the third image to each other, and the second mapping relationship may be used to map the positional relationship in the second image and the fourth image to each other. In some embodiments, step S290 may also include steps S291 to S295.

[0133] Step S291: Acquire first position information of the first target feature point in the third image, and acquire second position information of the second target feature point in the fourth image.

[0134] First, the first position information of the first target feature point in the third image can be obtained, and the second position information of the second target feature point in the fourth image can be obtained. It should be noted that the first target feature point and the second target feature point are the first target feature point and the second feature point that match each other in the matching feature pair.

[0135] Exemplarily, taking the third image as an example, a third image coordinate system can be constructed based on the third image. For example, the third image coordinate system is constructed with the upper left corner vertex of the third image as the origin, the first side of the third image as the square of the x-axis, and the second side of the third image as the positive direction of the y-axis. Then, the first position information of each first target feature point can be determined in the third image coordinate system, for example, each first position information can be represented by a coordinate in the form of (x, y).

[0136] Similarly, in the fourth image, a fourth image coordinate system may be constructed, and then the second position information of each second target feature point may be determined in the fourth image coordinate system. For example, each second position information may also be represented by coordinates in the form of (x, y).

[0137] Optionally, since the positions of each first feature point in the third image and the positions of each second feature point in the first feature point set and the second feature point set obtained by the feature extraction model may be discrete, feature interpolation processing can also be performed on the position of each first feature point in the third image, and feature interpolation processing can be performed on the position of each second feature point in the fourth image to obtain the non-discrete position of the first feature point in the third image, and to obtain the non-discrete position of the second feature point in the fourth image.

[0138] Step S292: Search for a pixel point in the first image that corresponds to a position having a first mapping relationship with the first position information as a first target pixel point, and search for a pixel point in the second image that corresponds to a position having a second mapping relationship with the second position information as a second target pixel point.

[0139] As can be seen from the above introduction, the first image and the third image may have a first mapping relationship, and the second image and the fourth image may have a second mapping relationship. Therefore, a pixel point corresponding to a position having a first mapping relationship with the first position information may be searched in the first image as the first target pixel point.

[0140] Similar to the third image coordinate system, the first image coordinate system can also be constructed based on the first image, so that the first mapping relationship can be determined based on the first image coordinate system and the third image coordinate system. Then, a pixel point corresponding to a position having a first mapping relationship with the first position information is searched in the first image as the first target pixel point.

[0141] It should be noted that the pixel point corresponding to the position having a first mapping relationship with the first position information is the third position information corresponding to the first position information determined in the first image through the first mapping relationship, and the pixel point corresponding to the third position information in the first image is the first target pixel point.

[0142] Similarly, a second image coordinate system can also be constructed based on the second image, so that a second mapping relationship can be determined based on the second image coordinate system and the fourth image coordinate system. Then, a pixel point corresponding to a position having a second mapping relationship with the second position information is searched in the second image as a second target pixel point.

[0143] Step S293: obtaining a first number of the first target pixel points in each row of pixel points in the first image, and obtaining a second number of the second target pixel points in each row of pixel points in the second image.

[0144] Step S294: Determine the row of pixel points corresponding to the largest first number in the first image as the first matching area.

[0145] Step S295: Determine the row of pixel points corresponding to the largest second number in the second image as the second matching area.

[0146] Further, after acquiring the first target pixel points included in the first image and the second target pixel points included in the second image, a matching area may be determined based on the first target pixel points and the second target pixel points.

[0147] In some embodiments, a first number of the first target pixels in each row of pixels in the first image may be obtained, and a second number of the second target pixels in each row of pixels in the second image may be obtained.

[0148] It is understandable that the pixels in the first image may include the first target pixels and the pixels other than the first target pixels; similarly, the pixels in the second image may include the second target pixels and the pixels other than the second target pixels. Therefore, the first number of pixels in each row of pixels in the first image including the first target pixels may be first obtained; and the second number of pixels in each row of pixels in the second image including the second target pixels may be obtained.

[0149] Thus, the row of pixel points corresponding to the largest first number in the first image can be determined as the first matching area; and the row of pixel points corresponding to the largest second number in the second image can be determined as the second matching area.

[0150] Optionally, taking the scrolling operation direction as scrolling down as an example, the first ordinate of the first target pixel point in the first image coordinate system is generally greater than or equal to the second ordinate of the second target pixel point corresponding to the first target pixel point in the second image. Therefore, the first target pixel point and the second target pixel point can also be screened based on the first ordinate of the first target pixel point in the first image coordinate system and the second ordinate of the second target pixel point in the second image.

[0151] Exemplarily, the first ordinate determined by the mutually matching first target feature points corresponding to each matching feature pair and the second ordinate determined by the second target feature points can be regarded as a pair of mutually matching first ordinates and second ordinates.

[0152] Thus, among the mutually matching first ordinates and second ordinates, it is possible to search for mutually matching target ordinates whose first ordinates are smaller than the second ordinates, and then exclude the first target pixel points corresponding to the target ordinates from the first target pixel points, and exclude the second target pixel points corresponding to the target ordinates from the second target pixel points, so as to implement a screening operation on the first target pixel points and the second target pixel points. Subsequently, the first matching area and the second matching area are determined based on the first target pixel points and the second target pixel points after the screening operation.

[0153] Optionally, the first target feature points and the second target feature points in each matching feature pair can also be screened based on the set. For example, when determining the matching feature pairs in the aforementioned steps, the first target feature points and the second target feature points can also be divided to generate a plurality of first sets including a plurality of first target feature points, and a plurality of second sets including a plurality of second target feature points. That is, each first set includes at least part of the first target feature points, and each second set includes at least part of the second target feature point set. It should be noted that, similar to the matching feature pairs, each first set can be matched with a second set.

[0154] Thus, in an embodiment of the present application, the third number of first target feature points included in each first set can be obtained, and then the first target set corresponding to the largest third number and the second target set that matches the first target set can be found to determine the first matching area and the second matching area based on the first target feature points corresponding to the first target set and the second target feature points corresponding to the second target set.

[0155] By screening the first target feature points and the second target feature points in the matching feature pair, the amount of calculation required to determine the first matching area and the second matching area can be reduced, and the accuracy of determining the first matching area and the second matching area can be improved to a certain extent.

[0156] Step S2100: splicing the first image and the second image based on the first matching area and the second matching area to obtain the target image.

[0157] Then, the first image and the second image may be spliced ​​based on the first matching area and the second matching area to obtain the target image.

[0158] Exemplarily, the target image may be obtained by splicing an image area of ​​the first matching area in the first image behind the direction of the scrolling operation and an image area of ​​the second matching area in the second image before the direction of the scrolling operation.

[0159] The description will continue with the example of scrolling down. The image area above the first matching area in the first image and the image area below the second matching area in the second image may be spliced ​​to obtain the target image. Thus, in the screenshot scenario, the screenshot image is obtained, that is, the target image is the screenshot image.

[0160] In the image stitching method provided in the embodiment of the present application, the first twin network and the second twin network can share parameters and weights by constructing a twin network, without training or storing two independent sets of parameters and weights, which can reduce the demand for storage space. In addition, the first twin network and the second twin network can perform feature point extraction in parallel, thereby effectively improving the overall efficiency of image stitching. Furthermore, in the embodiment of the present application, the feature points are screened by the confidence score of each feature point, and the first target feature point and the second target feature point can be directly matched, which reduces the processing time required for matching, and also improves the matching accuracy to a certain extent, that is, improves the accuracy of the obtained matching feature pair. Further, through the aforementioned operation of screening the first target feature point and the second target feature point, the amount of calculation required to determine the first matching area and the second matching area can be reduced, and the accuracy of determining the first matching area and the second matching area can be improved to a certain extent. In addition, the feature extraction model can also be obtained by end-side model transplantation, quantization processing, and parallel acceleration processing. Thus, the inference rate of the feature extraction model deployed to the electronic device can be improved, and the demand for processing power and storage capacity of the feature extraction model can also be reduced.

[0161] See also Figure 5 , Figure 5A method flow chart of an image stitching method provided in an embodiment of the present application is shown. The image stitching method can be applied to Figure 1 The electronic device in the image stitching scene 100 shown in the figure may specifically use a processor of the electronic device as an execution subject for executing the image stitching method. The image stitching method may include steps S310 to S3130.

[0162] Step S310: Acquire a first image.

[0163] Initially, a first image may be acquired.

[0164] Step S320: cropping the first image.

[0165] A first cropping area may be determined in the first image, and then an image other than the first cropping area in the first image may be obtained to achieve cropping of the first image.

[0166] Step S330: downsampling.

[0167] Furthermore, the image except the first cropping area in the first image may be down-sampled to obtain a third image.

[0168] Step S340: feature point extraction.

[0169] Then, feature point extraction may be performed on the third image to obtain a first feature point set.

[0170] Step S350: screening the first feature point set.

[0171] The first feature point set may be further screened to obtain first target feature points that meet the target condition.

[0172] Step S360: Acquire a second image.

[0173] Step S370: cropping the second image.

[0174] Step S380: downsampling.

[0175] Step S390: feature point extraction.

[0176] Step S3100: Screening the second feature point set.

[0177] Among them, steps S360 to S3100 are similar to steps S310 to S350, and second target feature points that meet the target conditions can be obtained.

[0178] Step S3110: Obtain matching feature pairs.

[0179] Then, a matching feature pair may be determined based on the first target feature points and the second target feature points obtained after screening.

[0180] Step S3120: Screening matching feature pairs.

[0181] The matching feature pairs can be further screened. It should be noted that the screening of the matching feature pairs is essentially the screening operation of the first target feature points and the second target feature points in each matching feature pair. The third number of first target feature points included in each first set can be obtained, and then the first target set corresponding to the largest third number and the second target set that matches the first target set can be found to determine the first matching area and the second matching area based on the first target feature points corresponding to the first target set and the second target feature points corresponding to the second target set.

[0182] Step S3130: stitching to obtain the target image.

[0183] Then, the first image and the second image may be spliced ​​based on the first matching area and the second matching area to obtain the target image.

[0184] For a detailed description of each of the above steps, please refer to the description of the steps in the above embodiments, which will not be repeated here.

[0185] In addition, the inventors also verified the effect of the image stitching method provided in the embodiment of the present application through experiments. Specifically, the inventors used a total of 1320 groups of test images in the experiment, wherein each group of test images includes a first image and a second image obtained by two consecutive screenshot operations of the electronic device when the screen of the electronic device is scrolled. Please refer to Table 1 below for detailed experimental data.

[0186] Table 1

[0187]

[0188] According to the experimental results in Table 1 above, it is estimated that compared with the existing image stitching method, the image stitching method provided by the embodiment of the present application can achieve a performance improvement of about 70%.

[0189] Among them, splicing problems may include duplication or truncation of floating icons, misalignment caused by video and animation playback, inability to splice due to screen rebound, the presence of a large amount of repeated similar content on the page, splicing misalignment caused by animation changes or module sliding, and truncation of unloaded images due to network reasons.

[0190] For example, see Figure 6 , for example, see Figure 6 , Figure 6 A schematic diagram comparing the results of the image stitching method provided in the embodiment of the present application and the current image stitching method is shown.

[0191] Figure 6 The image 610 shown in FIG. 6 is the target image obtained by the image stitching method provided by the embodiment of the present application. Figure 6 Also shown in FIG. 6 is an image 620 , where the image 620 is a stitched image obtained by the current image stitching method.

[0192] In the image 620, a problem area 621 is marked by a box, wherein the problem area 621 is the area where there is a stitching problem in the stitched image. Figure 6 There is a problem that the floating icon is repeated or truncated in the image 620.

[0193] For example, see Figure 7 , Figure 7 A schematic diagram comparing the results of the image stitching method provided in the embodiment of the present application and the current image stitching method is shown.

[0194] Figure 7 The image 710 shown in FIG. 7 is the target image obtained by the image stitching method provided by the embodiment of the present application. Figure 7 Also shown in FIG. 7 is an image 720 , where the image 720 is a stitched image obtained by the current image stitching method.

[0195] In the image 720, a problem area 721 is marked by a box, wherein the problem area 721 is the area where there is a stitching problem in the stitched image. Figure 7 Image 720 in the video cannot be spliced ​​due to screen rebound.

[0196] For example, see Figure 8 , for example, see Figure 8 , Figure 8 A schematic diagram comparing the results of the image stitching method provided in the embodiment of the present application and the current image stitching method is shown.

[0197] Figure 8 The image 810 shown in FIG. 8 is the target image obtained by the image stitching method provided by the embodiment of the present application. Figure 8 Also shown in FIG. 8 is an image 820 , where the image 820 is a stitched image obtained by the current image stitching method.

[0198] In the image 820, a problem area 821 is marked by a box, wherein the problem area 821 is the area where there is a stitching problem in the stitched image. Figure 8 Image 820 in the image is truncated because the network is not loaded, resulting in incomplete display.

[0199] For example, see Fig. 9 , for example, see Fig. 9 , Fig. 9 A schematic diagram comparing the results of the image stitching method provided in the embodiment of the present application and the current image stitching method is shown.

[0200] Fig. 9 The image 910 shown in FIG. 1 is the target image obtained by the image stitching method provided by the embodiment of the present application. Fig. 9 Also shown in FIG. 9 is an image 920 , where image 920 is a stitched image obtained by the current image stitching method.

[0201] In the image 920, a problem area 921 is marked by a box, wherein the problem area 921 is the area where there is a stitching problem in the stitched image. Fig. 9 Image 920 in the page has a problem of having a large amount of repeated similar content.

[0202] In the above Figures 6 to 9 The images 610, 710, 810 and 910 in the figure are all target images obtained by the image stitching method provided by the embodiment of the present application, and there is no stitching problem.

[0203] See also Fig.10 , Fig.10 The structure block diagram of an image stitching device provided in an embodiment of the present application is shown. The image stitching device 1000 includes: an acquisition unit 1010, a size adjustment unit 1020, a feature point extraction unit 1030, a feature matching unit 1040 and a stitching unit 1050.

[0204] The acquisition unit 1010 is used to acquire a first image and a second image obtained by two screenshots of the electronic device in response to a user's screenshot operation on the electronic device.

[0205] The size adjustment unit 1020 is used to downsample the first image and the second image to obtain a third image corresponding to the first image and a fourth image corresponding to the second image.

[0206] Optionally, the resizing unit 1020 can also be used to determine a first cropping area in the first image and a second cropping area in the second image; downsample the image in the first image except the first cropping area as the third image, and downsample the image in the second image except the second cropping area as the fourth image.

[0207] Optionally, the resizing unit 1020 can also be used to determine a first side line corresponding to the first side in the first image; determine a first cropping area in the first image based on the first side line along the first direction; determine a second side line corresponding to the second side in the second image; and determine a second cropping area in the second image based on the second side line along the opposite direction of the first direction.

[0208] The feature point extraction unit 1030 is used to extract feature points that meet preset feature point conditions from the third image and the fourth image to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image.

[0209] Optionally, the feature point extraction unit 1030 can also be used to extract feature points of the third image that meet preset feature point conditions through the first twin network branch in the feature extraction model to obtain a first feature point set corresponding to the third image; and to extract feature points of the fourth image that meet preset feature point conditions through the second twin network branch in the feature extraction model to obtain a second feature point set corresponding to the fourth image, wherein the first twin network branch and the second twin network branch share parameters.

[0210] Optionally, the feature point extraction unit 1030 can also be used to obtain a first model for extracting features from an image; perform end-side model transplantation and quantization processing on the first model to obtain a second model; and perform parallel acceleration processing on the second model to obtain the feature extraction model.

[0211] The feature matching unit 1040 is configured to determine at least one matching feature pair based on the first feature point set and the second feature point set, wherein each matching feature pair includes a first target feature point and a second target feature point that match each other.

[0212] Optionally, the feature matching unit 1040 can also be used to perform screening operations on the first feature point set and the second feature point set, respectively, to obtain first target feature points and second target feature points that meet the target conditions; determine the similarity between each first target feature point and each second target feature point; determine the second target feature point with the highest similarity corresponding to each first target feature point as the second target feature point that matches the first target feature point, so as to obtain at least one matching feature pair.

[0213] Optionally, the feature matching unit 1040 can also be used to obtain a first confidence score for each of the first feature points and a second confidence score for each of the second feature points; filter out a specified number of first feature points corresponding to the highest first confidence scores as first target feature points that meet the target conditions; filter out a specified number of second feature points corresponding to the highest second confidence scores as second target feature points that meet the target conditions.

[0214] The stitching unit 1050 is configured to stitch the first image and the second image together based on at least one of the matching feature pairs to obtain a target image.

[0215] Optionally, the stitching unit 1050 can also be used to determine a first matching area in the first image based on the matching feature pair, and to determine a second matching area in the second image; stitching the first image and the second image based on the first matching area and the second matching area to obtain the target image.

[0216] Optionally, the stitching unit 1050 can also be used to obtain the first position information of the first target feature point in the third image, and to obtain the second position information of the second target feature point in the fourth image; to search for a pixel point corresponding to a position having a first mapping relationship with the first position information in the first image as a first target pixel point, and to search for a pixel point corresponding to a position having a second mapping relationship with the second position information in the second image as a second target pixel point; to obtain a first number of the first target pixels in each row of pixels in the first image, and to obtain a second number of the second target pixels in each row of pixels in the second image; to determine the row of pixels corresponding to the largest first number in the first image as a first matching area; and to determine the row of pixels corresponding to the largest second number in the second image as a second matching area.

[0217] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0218] In several embodiments provided in the present application, the coupling between the units may be electrical, mechanical or other forms of coupling. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0219] See also Fig.11 , Fig.11 The block diagram of a structure of an electronic device provided in an embodiment of the present application is shown. The electronic device 110 may be a smart phone, a desktop computer, a vehicle-mounted computer, a server or a tablet computer, etc. The electronic device 110 in the present application may include one or more of the following components: a processor 111, a memory 112 and one or more application programs, wherein the processor 111 is electrically connected to the memory 112, and the one or more program configurations are used to execute the methods described in the aforementioned embodiments.

[0220] The processor 111 may include one or more processing cores. The processor 111 uses various interfaces and lines to connect various parts of the entire electronic device 110, and executes various functions and processes data of the electronic device 110 by running or executing instructions, programs, code sets or instruction sets stored in the memory 112, and calling data stored in the memory 112. Optionally, the processor 111 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 111 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and computer programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 111, but may be implemented separately through a communication chip. Specifically, the method described in the above embodiments may be executed by one or more processors 111 .

[0221] For some embodiments, the memory 112 may include a random access memory (RAM) or a read-only memory (ROM). The memory 112 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 112 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device 110 during use, etc.

[0222] See also Fig.12, which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1200 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0223] The computer readable storage medium 1200 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1200 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1200 has storage space for program code 1210 that executes any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 1210 can be compressed, for example, in an appropriate form.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image stitching method, characterized in that: include: In response to a user's screenshot operation on the electronic device, acquiring a first image and a second image obtained by two screenshots of the electronic device; Downsampling the first image and the second image to obtain a third image corresponding to the first image and a fourth image corresponding to the second image; Extracting feature points of the third image and the fourth image that meet preset feature point conditions to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image; Based on the first feature point set and the second feature point set, determining at least one matching feature pair, wherein each matching feature pair includes a first target feature point and a second target feature point that match each other; Based on at least one of the matching feature pairs, the first image and the second image are spliced ​​to obtain a target image.

2. The method according to claim 1, characterized in that The downsampling the first image and the second image to obtain a third image corresponding to the first image and a fourth image corresponding to the second image includes: determining a first cropping region in the first image, and determining a second cropping region in the second image; The image of the first image except the first cropping area is down-sampled to be used as the third image, and the image of the second image except the second cropping area is down-sampled to be used as the fourth image.

3. The method according to claim 2, characterized in that The electronic device includes a first side and a second side opposite to each other, screen display content corresponding to the first image and screen display content corresponding to the second image are sequentially distributed along a first direction, and determining a first cropping area in the first image and determining a second cropping area in the second image includes: determining a first edge line corresponding to the first side edge in the first image; determining a first cropping area in the first image along the first direction based on the first edge line; determining a second sideline corresponding to the second side in the second image; A second cropping area is determined in the second image based on the second edge line in a direction opposite to the first direction.

4. The method according to claim 1, characterized in that The determining at least one matching feature pair based on the first feature point set and the second feature point set includes: Performing screening operations on the first feature point set and the second feature point set respectively to obtain first target feature points and second target feature points that meet target conditions; Determining a similarity between each first target feature point and each second target feature point; A second target feature point with the highest similarity corresponding to each first target feature point is determined as the second target feature point that matches the first target feature point, so as to obtain at least one matching feature pair.

5. The method according to claim 4, characterized in that The filtering operation is performed on the first feature point set and the second feature point set respectively to obtain the first target feature point and the second target feature point that meet the target condition, including: Obtaining a first confidence score for each of the first feature points and a second confidence score for each of the second feature points; Filter out a specified number of first feature points corresponding to the highest first confidence scores as first target feature points that meet the target condition; A specified number of second feature points corresponding to the second highest confidence scores are screened out as second target feature points that meet the target condition.

6. The method according to claim 1, characterized in that The extracting feature points that meet preset feature point conditions from the third image and the fourth image to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image includes: Extracting feature points of the third image that meet preset feature point conditions through a first twin network branch in a feature extraction model to obtain a first feature point set corresponding to the third image; The feature points of the fourth image that meet the preset feature point conditions are extracted through the second twin network branch in the feature extraction model to obtain a second feature point set corresponding to the fourth image, wherein the first twin network branch and the second twin network branch share parameters.

7. The method according to claim 6, characterized in that Before extracting feature points of the third image that meet preset feature point conditions through the first twin network branch in the feature extraction model to obtain a first feature point set corresponding to the third image, the method further includes: Acquire a first model for performing feature extraction on an image; Performing end-side model transplantation and quantization processing on the first model to obtain a second model; The second model is processed in parallel and accelerated to obtain the feature extraction model.

8. The method according to claim 1, characterized in that The step of stitching the first image and the second image based on at least one of the matching feature pairs to obtain a target image includes: Determining a first matching region in the first image and a second matching region in the second image based on the matching feature pairs; The first image and the second image are spliced ​​based on the first matching area and the second matching area to obtain the target image.

9. The method according to claim 8, characterized in that The determining a first matching area in the first image based on the matching feature pair, and determining a second matching area in the second image, respectively, comprises: Acquire first position information of the first target feature point in the third image, and acquire second position information of the second target feature point in the fourth image; Searching in the first image for a pixel point corresponding to a position having a first mapping relationship with the first position information as a first target pixel point, and searching in the second image for a pixel point corresponding to a position having a second mapping relationship with the second position information as a second target pixel point; Acquire a first number of the first target pixels in each row of pixels in the first image, and acquire a second number of the second target pixels in each row of pixels in the second image; Determine the row of pixel points corresponding to the largest first number in the first image as a first matching area; The row of pixel points corresponding to the largest second number in the second image is determined as a second matching area.

10. An image stitching device, characterized in that: include: An acquisition unit, configured to acquire a first image and a second image obtained by two screenshots of the electronic device in response to a user's screenshot operation on the electronic device; a size adjustment unit, configured to downsample the first image and the second image to obtain a third image corresponding to the first image and a fourth image corresponding to the second image; a feature point extraction unit, configured to extract feature points of the third image and the fourth image that meet preset feature point conditions, to obtain a first feature point set corresponding to the third image and a second feature point set corresponding to the fourth image; A feature matching unit, configured to determine at least one matching feature pair based on the first feature point set and the second feature point set, wherein each matching feature pair includes a first target feature point and a second target feature point that match each other; A stitching unit is used to stitch the first image and the second image based on at least one matching feature pair to obtain a target image.

11. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1 to 9.

Images