Method, device and computer-readable storage medium for determining panoramic image jump position

Abstract

Embodiments of the present invention provide a method, device and computer-readable storage medium for determining a jump position of a panoramic image. The method for determining a jump position of a panoramic image according to an embodiment of the present invention includes: acquiring a first panoramic image and a second panoramic image having the same horizontal width; adjusting at least one of the first panoramic image and the second panoramic image according to the posture data of the camera that captured the first panoramic image and the second panoramic image; after the adjustment, performing feature extraction on the first panoramic image and the second panoramic image, and based on the result of the feature extraction, screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image to screen out at least one pair of first image columns; and determining the jump position of the panoramic image based on the horizontal position of the screened at least one pair of first image columns.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for determining a jump position of a panoramic image, and a computer-readable storage medium. Background Art

[0002] Virtual roaming is an important branch of virtual reality (VR) technology, involving many industries such as real estate, urban planning, tourism, aerospace, and medicine. Through the combination of virtual scene creation technology and virtual roaming technology, users can autonomously roam in three-dimensional scenes such as buildings, cities, or game scenes, thereby gaining an intuitive understanding of the above roaming scenes.

[0003] Generally speaking, virtual tour producers can use production tools such as 720 Cloud to process pre-shot panoramic images, select jump positions on two panoramic images with scene associations and set mutually linked guide points on them, thereby constructing jump relationships between different panoramic images to complete virtual tour production. In the actual virtual tour process, users can select and click on the guide points displayed in the current roaming scene, thereby leaving the current scene from the jump position on the current panoramic image, and automatically jump to the jump position on the next panoramic image to which the guide point is linked, so as to continue roaming the next scene.

[0004] However, the above method of manually selecting corresponding jump positions on two panoramic images and setting guide points is seriously inefficient, and if an inappropriate jump position is selected to set the guide point, the user's perspective of observing the three-dimensional scene will change significantly before and after the jump, affecting the user's actual roaming experience. It can be seen that a method and device that can automatically and accurately determine the jump position of the panoramic image so as to reasonably set the guide point is needed. Summary of the invention

[0005] To solve the above technical problems, according to one aspect of the present invention, a method for determining a panoramic image jump position is provided, comprising: acquiring a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in a three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; adjusting at least one of the first panoramic image and the second panoramic image according to posture data of cameras for shooting the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; after the adjustment, performing feature extraction on the first panoramic image and the second panoramic image, and based on the result of the feature extraction, screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image, so as to screen out at least one pair of first image columns whose feature similarity meets a predetermined requirement in each pair of image columns; and determining a panoramic image jump position for jumping between the first panoramic image and the second panoramic image based on the screened at least one pair of first image columns.

[0006] According to another aspect of the present invention, a panoramic image jump position determination device is provided, comprising: an acquisition unit, configured to acquire a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in a three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; an adjustment unit, configured to perform adjustment processing on at least one of the first panoramic image and the second panoramic image according to posture data of cameras for shooting the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; a feature extraction and screening unit, configured to perform feature extraction processing on the first panoramic image and the second panoramic image after the adjustment processing, and screen each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image based on the result of the feature extraction processing, so as to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets a predetermined requirement; and a position determination unit, configured to determine a panoramic image jump position for jumping between the first panoramic image and the second panoramic image based on the screened at least one pair of first image columns.

[0007] According to another aspect of the present invention, there is provided an image processing device, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: acquiring a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in a three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; performing adjustment processing on at least one of the first panoramic image and the second panoramic image according to posture data of cameras for shooting the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; after the adjustment processing, performing feature extraction processing on the first panoramic image and the second panoramic image, and based on the result of the feature extraction processing, screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image, so as to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets a predetermined requirement; and determining a panoramic image jump position for jumping between the first panoramic image and the second panoramic image based on the screened at least one pair of first image columns.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the following steps when executed by a processor: acquiring a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in a three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; performing adjustment processing on at least one of the first panoramic image and the second panoramic image according to posture data of cameras for shooting the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; after the adjustment processing, performing feature extraction processing on the first panoramic image and the second panoramic image, and based on the result of the feature extraction processing, screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image, so as to screen out at least one pair of first image columns whose feature similarity meets a predetermined requirement in each pair of image columns; and determining a panoramic image jump position for jumping between the first panoramic image and the second panoramic image based on the screened at least one pair of first image columns.

[0009] According to the above-mentioned method, device and computer-readable storage medium for determining the jump position of a panoramic image of the present invention, it is possible to find at least one pair of image columns whose feature similarity meets predetermined requirements on two panoramic images, so as to determine the jump position for jumping between the two panoramic images based on the screened image columns and set the guide point. This method, device and computer-readable storage medium for automatically determining the jump position of a panoramic image can automatically and accurately determine the jump position of a panoramic image so as to reasonably set the guide point, improve the efficiency of setting the guide point, and can keep the user's perspective unchanged before and after the jump, so that the user seems to jump to the next scene by actually moving forward, giving the user a better progressive roaming experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.

[0011] Figure 1 A method for determining a jump position of a panoramic image according to an embodiment of the present invention is shown;

[0012] Figure 2 A schematic spatial plane diagram of two adjacent subspaces according to an embodiment of the present invention and an example of a first panoramic image and a second panoramic image obtained by photographing at two mutually visible photographing points in the two adjacent subspaces;

[0013] Figure 3 An example of a first panoramic image and a second panoramic image after adjustment processing is performed on the second panoramic image according to an embodiment of the present invention is shown;

[0014] Figure 4 An example of a matching result obtained by performing feature point matching on a first panoramic image and a second panoramic image according to an embodiment of the present invention is shown;

[0015] Figure 5 An example of filtering each pair of image columns on the first panoramic image and the second panoramic image by filtering each pair of feature points according to an embodiment of the present invention is shown;

[0016] Figure 6 An example of implementing screening of each pair of image columns on the first panoramic image and the second panoramic image by screening each pair of image column features according to an embodiment of the present invention is shown;

[0017] Figure 7 An example of determining another pair of first image columns according to the horizontal width of a panoramic image when only one pair of first image columns is screened out according to one embodiment of the present invention is shown;

[0018] Figure 8 An example of performing aggregation processing on image columns when more than two pairs of first image columns are screened out according to an embodiment of the present invention is shown;

[0019] Fig. 9 An example of determining a departure position and an arrival position for jumping between panoramic images based on two pairs of first image columns according to an embodiment of the present invention is shown;

[0020] Fig.10 A block diagram showing an image processing apparatus according to an embodiment of the present invention;

[0021] Fig.11 A block diagram of an image processing apparatus according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] The following will describe a method, device, and computer-readable storage medium for determining a jump position of a panoramic image according to an embodiment of the present invention with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements throughout. It should be understood that the embodiments described herein are merely illustrative and should not be interpreted as limiting the scope of the present invention.

[0023] The following will refer to Figure 1 A method for determining a jump position of a panoramic image according to an embodiment of the present invention is described. Figure 1 A flow chart of the panoramic image jump position determination method 100 is shown.

[0024] like Figure 1 As shown, in step S101, a first panoramic image and a second panoramic image having the same horizontal width are acquired, wherein a shooting point for shooting the first panoramic image is located in the three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image.

[0025] In this step, the three-dimensional space corresponding to the panoramic image is the three-dimensional space where all objects displayed by the panoramic image are located.

[0026] In one example, the first panoramic image and the second panoramic image can be obtained by shooting at two shooting points located in the same subspace (for example, in the same room of the whole house). In this case, the two panoramic images captured by the above shooting point selection method at least correspond to the same subspace (that is, the two panoramic images at least show the objects in the same subspace), and the two shooting points are located in the same subspace.

[0027] In another example, shooting may also be performed at two shooting points located in adjacent subspaces (for example, in two adjacent rooms of a whole house) to obtain a first panoramic image and a second panoramic image, as long as a line connecting the two shooting points is not blocked or shielded by an intermediate structure such as a door or a wall, so that the panoramic cameras at the two shooting points are visible to each other, thereby obtaining the first panoramic image and the second panoramic image according to an embodiment of the present invention.

[0028] Figure 2 The schematic diagram of the spatial plane of two adjacent subspaces according to one embodiment of the present invention and an example of a first panoramic image and a second panoramic image obtained by shooting at two mutually visible shooting points in the two adjacent subspaces are shown. The first panoramic image above is obtained by shooting at shooting point 1 in the first subspace, and the second panoramic image below is obtained by shooting at shooting point 2 in the second subspace. Figure 2 As shown, since the panoramic cameras at the two shooting points are visible to each other, the first panoramic image includes not only the image portion corresponding to the first subspace, but also the image portion corresponding to sub-region 2 where shooting point 2 is located in the second subspace (i.e., image portion 201), and the second panoramic image includes not only the image portion corresponding to the second subspace, but also the image portion corresponding to sub-region 1 where shooting point 1 is located in the first subspace (i.e., image portion 202). In addition, as Figure 2 As further shown in , the image portion 201 in the first panoramic image and the left half of the second panoramic image both correspond to the second subspace, and both display objects such as a dark conference table and a television arranged in the second subspace, and therefore have a large degree of feature similarity. Correspondingly, the image portion 202 in the second panoramic image and the left half of the first panoramic image both correspond to the first subspace, and both display objects such as a light conference table, a television, and plants next to the television arranged in the first subspace, and therefore have a large degree of feature similarity.

[0029] In the following, the specific processing for screening out at least one pair of image columns whose feature similarity meets predetermined requirements on the first panoramic image and the second panoramic image will be described in detail, so as to determine a jump position for jumping between the two panoramic images based on the screened image columns.

[0030] In step S102, at least one of the first panoramic image and the second panoramic image is adjusted according to the posture data of the camera that shoots the first panoramic image and the second panoramic image to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture.

[0031] Figure 3An example of a first panoramic image and a second panoramic image after adjustment processing of the second panoramic image according to an embodiment of the present invention is shown. Figure 3 As shown, the above adjustment process adjusts the image part 202 in the second panoramic image that has a greater degree of feature similarity with the left half of the first panoramic image to the left half of the image, and adjusts the left half of the second panoramic image that has a greater degree of feature similarity with the image part 201 in the first panoramic image to the right half of the image. Through the above adjustment process, at least one pair of image columns whose degree of feature similarity on the first panoramic image and the second panoramic image meets the predetermined requirements can be aligned in the horizontal direction, so as to facilitate the subsequent direct calculation of the degree of feature similarity of each pair of image columns with the same horizontal position and screening, so as to reduce the amount of calculation required for the related processing and improve the accuracy of image column screening.

[0032] The camera posture in this step is the orientation of the camera in three-dimensional space when shooting the panoramic image. In this step, the posture data of the camera when shooting the first panoramic image and the second panoramic image can be collected using a posture sensor on the camera, for example, using a gyroscope on the camera to collect angular velocity meter parameters. In addition, other means can also be used to obtain the posture data of the camera when shooting the first panoramic image and the second panoramic image, which are not limited here.

[0033] After acquiring the camera's posture data, at least one of the first panoramic image and the second panoramic image can first be projected into a three-dimensional spherical image through a spherical projection transformation, and the difference between the camera's posture data when capturing the two panoramic images can be used to calculate the angle by which the three-dimensional spherical image needs to be rotated to perform a rotation operation on it, and finally the rotated three-dimensional spherical image is reprojected back to a planar panoramic image through a spherical projection inverse transformation to obtain an adjusted panoramic image. It should be noted that those skilled in the art can use other means besides the above-mentioned projection transformation and rotation operation to adjust at least one of the first panoramic image and the second panoramic image, and obtain the first panoramic image and the second panoramic image corresponding to the same camera posture, which is not limited here.

[0034] It should be noted here that since the camera may not be completely horizontal during the actual shooting process, the panoramic image obtained by shooting may be skewed to a certain extent. At this time, before or after the above-mentioned adjustment process is performed on the first panoramic image and the second panoramic image, the two panoramic images may be vertically corrected using the camera's posture data collected by the above-mentioned posture sensor to improve the accuracy of subsequent processing. In addition, in another example, straight lines may be detected on the panoramic image obtained by shooting and six vanishing points may be calculated through Hough transform, thereby obtaining three vanishing directions as the camera's posture data, and using them to perform vertical correction processing on the two panoramic images.

[0035] In step S103, after the adjustment process, feature extraction processing is performed on the first panoramic image and the second panoramic image, and based on the result of the feature extraction processing, each pair of image columns with the same horizontal position on the first panoramic image and the second panoramic image is screened to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets predetermined requirements.

[0036] Two screening schemes for screening at least one pair of first image columns whose feature similarity on the first panoramic image and the second panoramic image meets a predetermined requirement, which are used in the present invention, are described in detail below. However, it should be noted that the present invention is not limited thereto, and in actual application, various other means may be used to screen each pair of image columns on the first panoramic image and the second panoramic image, so as to screen at least one pair of image columns with the highest feature similarity as at least one pair of first image columns according to an embodiment of the present invention.

[0037] Image column screening solution 1: feature point matching

[0038] In general, this solution implements the screening of each pair of image columns on the first panoramic image and the second panoramic image by matching feature points on the first panoramic image and the second panoramic image, and screening each matched pair of feature points.

[0039] In this solution, the feature extraction processing of the first panoramic image and the second panoramic image may include: performing feature point matching on the first panoramic image and the second panoramic image to extract each pair of feature points matched on the first panoramic image and the second panoramic image.

[0040] Figure 4 An example of a matching result obtained by performing feature point matching on a first panoramic image and a second panoramic image according to an embodiment of the present invention is shown. Figure 4In the figure, each pair of feature points matched on the first panoramic image and the second panoramic image is framed by a black dotted frame. In the process of feature point matching, feature point matching processing can be performed on the complete first panoramic image and the second panoramic image to directly extract each pair of feature points matched on the first panoramic image and the second panoramic image. In addition, in order to further improve the accuracy of feature point matching, the first panoramic image and the second panoramic image can also be divided into a plurality of image blocks with the same horizontal width in the horizontal direction, and feature point matching processing can be performed on each pair of image blocks with the same horizontal position on the first panoramic image and the second panoramic image, and finally the feature point matching results of each pair of image blocks are reproduced on the complete first panoramic image and the second panoramic image to extract each pair of feature points matched on the first panoramic image and the second panoramic image. It should be noted here that the above-mentioned feature point matching processing can be implemented by methods such as Oriented FAST and Rotated BRIEF (ORB), Speeded Up Robust Features (SURF), Scale Invariant Feature Transform (SIFT), etc., which are well known to those skilled in the art, and the present invention is not limited thereto.

[0041] After the above-mentioned feature point matching processing, each matched pair of feature points can be used as the result of the feature extraction processing, and the screening of each pair of image columns with the same horizontal position on the first panoramic image and the second panoramic image based on the result of the feature extraction processing to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets predetermined requirements may include: calculating the position difference between the horizontal positions of each pair of feature points in each matched pair of feature points, and screening out at least one pair of first feature points in each matched pair of feature points whose position difference is less than a first threshold value, so as to screen out at least one pair of first image columns corresponding to the horizontal position of the at least one pair of first feature points.

[0042] Specifically, the difference between the horizontal positions (i.e., horizontal coordinate positions, in pixels) of each pair of feature points on the first panoramic image and the second panoramic image may be first calculated as the position difference of each pair of feature points. Subsequently, each calculated position difference may be compared with a preset first threshold, and at least one pair of first feature points whose position difference is less than the first threshold may be screened out. Figure 5 An example of filtering each pair of image columns on the first panoramic image and the second panoramic image by filtering each pair of feature points according to an embodiment of the present invention is shown. Figure 5As shown, through the above screening process, at least a pair of first feature points 501 and 502 whose position difference meets the first threshold requirement are screened out, and it can be seen that the horizontal positions of the pair of first feature points 501 and 502 are very close.

[0043] In order to further select at least one pair of first image columns corresponding to the horizontal position of the at least one pair of first feature points based on the selected at least one pair of first feature points, in one example, the positions of the corresponding pair of first image columns can be determined based on the average horizontal position of each pair of first feature points selected. Figure 5 As shown, the average of the horizontal positions of the first feature points 501 and 502 can be calculated to obtain their average horizontal positions, and a pair of image columns 503 and 504 located at the average horizontal position can be screened out on the first panoramic image and the second panoramic image to serve as a pair of first image columns corresponding to the horizontal positions of the pair of first feature points 501 and 502. In addition, in another example, a corresponding pair of first image columns can be screened out based on the horizontal position of one of the feature points in each pair of screened out first feature points (for example, a pair of image columns located at the horizontal position of the feature point 501 or the feature point 502 can be screened out on the first panoramic image and the second panoramic image) to serve as a pair of first image columns corresponding to the horizontal positions of the pair of first feature points, which is not limited here.

[0044] It should be noted that there is a situation where the pre-set first threshold is too large, which makes it impossible to screen out any pair of first feature points that meet the first threshold requirement and thus the first image column. Therefore, when the position difference of each matched pair of feature points is greater than the first threshold, the first threshold can be lowered until at least one pair of first feature points with a position difference less than the first threshold is screened out from each matched pair of feature points.

[0045] The at least one pair of first image sequences obtained by the above-mentioned feature point matching and screening scheme is the at least one pair of first image sequences whose feature similarity meets the predetermined requirement. In the following, another scheme for screening out at least one pair of first image sequences whose feature similarity meets the predetermined requirement on the first panoramic image and the second panoramic image will be described.

[0046] Image column screening solution 2: Using convolutional neural network to obtain feature maps

[0047] In general, this scheme filters each pair of image columns on the first panoramic image and the second panoramic image by sending the first panoramic image and the second panoramic image into a convolutional neural network (CNN) to obtain a corresponding feature map, and filtering each pair of image column features on the feature map.

[0048] In this solution, the feature extraction processing of the first panoramic image and the second panoramic image may include: inputting the first panoramic image and the second panoramic image into a convolutional neural network to obtain at least one first feature map of the first panoramic image and at least one second feature map of the second panoramic image having the horizontal width of the panoramic image. As is well known to those skilled in the art, by inputting an image into a convolutional neural network, at least one feature map of the image can be extracted through the convolutional layer of the convolutional neural network, wherein the at least one feature map includes a plurality of feature values ​​arranged corresponding to the positions of the features on the image, and the size of the feature map can be controlled by a zero-padding operation. In this solution, by controlling the size of the feature map, at least one first feature map of the first panoramic image and at least one second feature map of the second panoramic image having the same horizontal width as the first panoramic image and the second panoramic image can be obtained.

[0049] Figure 6 An example of implementing the filtering of each pair of image columns on the first panoramic image and the second panoramic image by filtering each pair of image column features according to an embodiment of the present invention is shown. Figure 6 , three first feature maps and three second feature maps extracted for the first panoramic image and the second panoramic image respectively are shown, but the present invention does not limit the number of the first feature maps and the second feature maps (i.e., the depth of the feature maps), nor does it limit the height of the first feature maps and the second feature maps in the vertical direction.

[0050] After obtaining the above-mentioned first feature map and second feature map, the first feature map and second feature map can be used as the results of the feature extraction processing, and the screening of each pair of image columns with the same horizontal position on the first panoramic image and the second panoramic image based on the results of the feature extraction processing to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets predetermined requirements includes: calculating the feature difference between each pair of image column features with the same horizontal position on the first feature map and the second feature map, and screening out at least one pair of first image column features in each pair of image column features whose feature difference is less than a second threshold value, so as to screen out at least one pair of first image columns corresponding to the horizontal position of the at least one pair of first image column features.

[0051] Specifically, image column features at each horizontal position can be extracted by column (for example, from left to right or from right to left in the horizontal direction) on the first feature map and the second feature map, wherein each extracted image column feature is a feature matrix including one or more feature vectors. After extracting each image column feature, the feature difference between each pair of image column features having the same horizontal position on the first feature map and the second feature map can be calculated. In one example, in order to calculate the feature difference between a pair of image column features, the distances (such as L2 distance, cosine similarity, etc.) between each feature vector having the same vertical position can be calculated in sequence in the vertical direction from top to bottom or from bottom to top for the pair of image column features, and each distance is summed up as the difference between the pair of image column features. However, it should be noted that those skilled in the art can use other means for calculating the difference between feature matrices to determine the feature difference between each pair of image column features according to an embodiment of the present invention, which is not limited here.

[0052] After obtaining the feature differences between each pair of image column features with the same horizontal position on the first feature map and the second feature map, the feature differences can be compared with the preset second threshold, thereby screening out at least one pair of first image column features with a feature difference less than the second threshold. Thereafter, based on the horizontal position of the at least one pair of image column features, at least one pair of image columns with the same horizontal position as the at least one pair of first image column features can be screened out on the first panoramic image and the second panoramic image as the at least one pair of first image columns with a feature similarity that meets the predetermined requirement. Figure 6 As shown, by performing feature difference calculation and threshold comparison on each pair of image column features on the first feature map and the second feature map, a pair of first image column features 601 and 602 and another pair of first image column features 603 and 604 whose feature difference is less than the second threshold are screened out, and accordingly, a pair of first image columns 605 and 606 having the same horizontal position as the first image column features 601 and 602 and another pair of first image columns 607 and 608 having the same horizontal position as the first image column features 603 and 604 are further screened out, wherein the above two pairs of first image columns are at least one pair of first image columns whose feature similarity meets predetermined requirements screened out on the first panoramic image and the second panoramic image.

[0053] It should be noted that there is a situation where the preset second threshold is too large, so that any pair of first image sequence features that meet the second threshold requirement cannot be screened out, and thus the first image sequence cannot be screened out. Therefore, in the case where the feature difference of each pair of image sequence features is greater than the second threshold, the second threshold can be lowered until at least one pair of first image sequence features whose feature difference is less than the second threshold is screened out from each pair of image sequence features.

[0054] Through the above two solutions, it is possible to screen out at least one pair of first image sequences whose feature similarity meets a predetermined requirement on the first panoramic image and the second panoramic image. Then, the present invention proceeds to the next step.

[0055] In step S104, a panoramic image jump position for jumping between the first panoramic image and the second panoramic image is determined based on the at least one pair of first image columns that are screened out.

[0056] Generally speaking, two jump positions need to be determined on a panoramic image, namely, an exit position for jumping from the panoramic image to another panoramic image, and an arrival position for jumping from the other panoramic image to the panoramic image. In the case where the number of the at least one pair of first image columns screened out is 2 and the two pairs of first image columns happen to be located in the left half and the right half of the panoramic image, respectively, the exit position on the first panoramic image and the arrival position on the second panoramic image for jumping from the first panoramic image to the second panoramic image, as well as the exit position on the second panoramic image and the arrival position on the first panoramic image for jumping from the second panoramic image to the first panoramic image can be determined directly based on the positions of the two pairs of first image columns screened out, thereby completing the selection of the jump positions of the panoramic images.

[0057] In addition, when the number of at least one pair of first image columns whose feature similarity meets the predetermined requirement is 1, the position of another pair of first image columns can be directly determined by using the horizontal width of the panoramic image. Specifically, when only one pair of first image columns is screened out, another pair of first image columns can be determined at a horizontal position that is half the horizontal width of the panoramic image away from the horizontal position of the screened pair of first image columns. Figure 7 An example of determining another pair of first image columns according to the horizontal width of a panoramic image when only one pair of first image columns is screened out according to an embodiment of the present invention is shown. In which, only a pair of first image columns 701 and 702 whose feature similarity meets a predetermined requirement are screened out on the first panoramic image and the second panoramic image. At this time, another pair of image columns 703 and 704 that are half the horizontal width of the panoramic image (i.e., W / 2) away from them can be directly determined as another pair of first image columns.

[0058] In addition, when the number of at least one pair of first image columns screened out whose feature similarity meets a predetermined requirement is greater than 2, the at least one pair of first image columns screened out may be aggregated to form two pairs of first image columns. Figure 8 FIG. 2 shows an example of performing aggregation processing on image columns when more than two pairs of first image columns are screened out according to an embodiment of the present invention. Figure 8As shown on the left side of , first image columns whose feature similarities greater than two pairs meet predetermined requirements are screened out on the first panoramic image and the second panoramic image, and after aggregation processing, a pair of aggregated first image columns 801 and 802 and another pair of first image columns 803 and 804 are finally obtained. Among them, the above-mentioned aggregation processing can be achieved by aggregating the horizontal positions of at least one pair of screened out first image columns by various means, for example, clustering the horizontal positions of each first image column to find their clustering center, and taking the image column located at the clustering center as the aggregated first image column, which is not limited here. It should be noted that those skilled in the art may not perform the above-mentioned aggregation processing, but directly select any two pairs of first image columns from the at least one pair of screened out first image columns, as long as the two pairs of first image columns are located in the left half and the right half of the panoramic image, respectively, which is not limited here.

[0059] Fig. 9 An example of determining a departure position and an arrival position for jumping between panoramic images based on two pairs of first image sequences according to an embodiment of the present invention is shown. Fig. 9 As shown in FIG. 1 , after the above steps, two pairs of image columns for setting jump positions are finally selected on the first panoramic image and the second panoramic image, namely, a pair of first image columns 901 and 902 located in the left half of the panoramic image and a pair of first image columns 903 and 904 located in the right half of the panoramic image. Fig. 9 As further shown in , the position of the image column 901 on the first panoramic image can be determined as the arrival position for jumping from the second panoramic image to the first panoramic image, the position of the image column 902 on the second panoramic image can be determined as the departure position for jumping from the second panoramic image to the first panoramic image, the position of the image column 903 on the first panoramic image can be determined as the departure position for jumping from the first panoramic image to the second panoramic image, and the position of the image column 904 on the second panoramic image can be determined as the arrival position for jumping from the first panoramic image to the second panoramic image, thereby completing the setting of the jump position. In addition, a guide point can be further set at any position of the above-mentioned image column to be displayed to the user and selected by the user during the virtual roaming process.

[0060] According to the above-mentioned panoramic image jump position determination method of the present invention, the panoramic image jump position can be automatically and accurately determined so as to reasonably set the guide point, improve the efficiency of setting the guide point, and make the user's perspective remain unchanged before and after the jump, so that the user seems to jump to the next scene by actually moving forward, giving the user a better progressive roaming experience.

[0061] Below, refer to Fig.10 The apparatus for determining a jump position of a panoramic image according to an embodiment of the present invention will be described below. Fig.10FIG. 1 is a block diagram of an image processing apparatus 1000 according to an embodiment of the present invention. Fig.10 As shown, the image processing device 1000 includes an acquisition unit 1010, an adjustment unit 1020, a feature extraction and screening unit 1030, and a position determination unit 1040. In addition to these units, the image processing device 1000 may also include other components, however, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here.

[0062] first, Fig.10 The acquisition unit 1010 in acquires a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in a three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image.

[0063] The three-dimensional space corresponding to the panoramic image is the three-dimensional space where all objects displayed by the panoramic image are located.

[0064] In one example, the first panoramic image and the second panoramic image can be obtained by shooting at two shooting points located in the same subspace (for example, in the same room of the whole house). In this case, the two panoramic images captured by the above shooting point selection method at least correspond to the same subspace (that is, the two panoramic images at least show the objects in the same subspace), and the two shooting points are located in the same subspace.

[0065] In another example, shooting may also be performed at two shooting points located in adjacent subspaces (for example, in two adjacent rooms of a whole house) to obtain a first panoramic image and a second panoramic image, as long as a line connecting the two shooting points is not blocked or shielded by an intermediate structure such as a door or a wall, so that the panoramic cameras at the two shooting points are visible to each other, thereby obtaining the first panoramic image and the second panoramic image according to an embodiment of the present invention.

[0066] The adjustment unit 1020 adjusts at least one of the first panoramic image and the second panoramic image according to the posture data of the camera that shoots the first panoramic image and the second panoramic image to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture.

[0067] Figure 3 An example of a first panoramic image and a second panoramic image after adjustment processing of the second panoramic image according to an embodiment of the present invention is shown. Figure 3As shown, the adjustment processing of the adjustment unit 1020 adjusts the image portion 202 in the second panoramic image having a greater degree of feature similarity with the left half of the first panoramic image to the left half of the image, and adjusts the left half of the second panoramic image having a greater degree of feature similarity with the image portion 201 in the first panoramic image to the right half of the image. Through the adjustment processing of the adjustment unit 1020, at least one pair of image columns whose degree of feature similarity on the first panoramic image and the second panoramic image meets a predetermined requirement can be aligned in the horizontal direction, so as to facilitate the subsequent direct calculation of the degree of feature similarity of each pair of image columns with the same horizontal position and screening, so as to reduce the amount of calculation required for the related processing and improve the accuracy of image column screening.

[0068] The camera posture is the orientation of the camera in three-dimensional space when shooting the panoramic image. The posture data of the camera when shooting the first panoramic image and the second panoramic image can be collected using a posture sensor on the camera, for example, using a gyroscope on the camera to collect angular velocity meter parameters. In addition, other means can be used to obtain the posture data of the camera when shooting the first panoramic image and the second panoramic image, which are not limited here.

[0069] After acquiring the camera's posture data, the adjustment unit 1020 can first project at least one of the first panoramic image and the second panoramic image into a three-dimensional spherical image through a spherical projection transformation, and use the difference between the camera's posture data when capturing the two panoramic images to calculate the angle by which the three-dimensional spherical image needs to be rotated to perform a rotation operation on it, and finally reproject the rotated three-dimensional spherical image back to a planar panoramic image through an inverse spherical projection transformation to obtain the adjusted panoramic image.

[0070] It should be noted here that since the camera may not be completely horizontal during the actual shooting process, the panoramic image obtained by shooting is skewed to a certain extent. At this time, the adjustment unit 1020 can use the camera posture data collected by the above-mentioned posture sensor to perform vertical correction processing on the two panoramic images before or after the above-mentioned adjustment processing on the first panoramic image and the second panoramic image, so as to improve the accuracy of subsequent processing. In addition, in another example, the adjustment unit 1020 can also detect straight lines on the panoramic image obtained by shooting and calculate six vanishing points through Hough transform, so as to obtain three vanishing directions as the camera posture data, and use it to perform vertical correction processing on the two panoramic images.

[0071] The feature extraction and screening unit 1030 performs feature extraction processing on the first panoramic image and the second panoramic image after the adjustment processing, and screens each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image based on the result of the feature extraction processing, so as to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets predetermined requirements.

[0072] Two screening schemes for screening at least one pair of first image columns whose feature similarity on the first panoramic image and the second panoramic image meets a predetermined requirement, which are used in the present invention, are described in detail below. However, it should be noted that the present invention is not limited thereto, and in actual application, various other means may be used to use the feature extraction and screening unit 1030 to screen each pair of image columns on the first panoramic image and the second panoramic image, so as to screen at least one pair of image columns with the highest feature similarity as at least one pair of first image columns according to an embodiment of the present invention.

[0073] Image column screening solution 1: feature point matching

[0074] In general, this solution implements filtering of each pair of image columns on the first panoramic image and the second panoramic image by performing feature point matching on the first panoramic image and the second panoramic image by the feature extraction and screening unit 1030 and screening each matched pair of feature points.

[0075] In this solution, the feature extraction and screening unit 1030 may perform feature extraction processing on the first panoramic image and the second panoramic image, which may include: performing feature point matching on the first panoramic image and the second panoramic image to extract each pair of feature points matched on the first panoramic image and the second panoramic image.

[0076] Figure 4 An example of a matching result obtained by performing feature point matching on a first panoramic image and a second panoramic image according to an embodiment of the present invention is shown. Figure 4, each pair of feature points matched on the first panoramic image and the second panoramic image are framed with a black dotted frame. In the process of feature point matching, the feature extraction and screening unit 1030 can perform feature point matching processing on the complete first panoramic image and the second panoramic image to directly extract each pair of feature points matched on the first panoramic image and the second panoramic image. In addition, in order to further improve the accuracy of feature point matching, the feature extraction and screening unit 1030 can also first divide the first panoramic image and the second panoramic image into a plurality of image blocks with the same horizontal width along the horizontal direction, and perform feature point matching processing on each pair of image blocks with the same horizontal position on the first panoramic image and the second panoramic image, and finally reproduce the feature point matching results of each pair of image blocks on the complete first panoramic image and the second panoramic image to extract each pair of feature points matched on the first panoramic image and the second panoramic image. It should be noted here that the above-mentioned feature point matching processing can be implemented using methods well known to those skilled in the art, such as Oriented FAST and Rotated BRIEF (ORB), Speeded Up Robust Features (SURF), Scale Invariant Feature Transform (SIFT), etc., and the present invention is not limited to this.

[0077] After the above-mentioned feature point matching processing, the feature extraction and screening unit 1030 can use the matched pairs of feature points as the results of the feature extraction processing, and the screening of the pairs of image columns with the same horizontal position on the first panoramic image and the second panoramic image based on the results of the feature extraction processing to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets the predetermined requirements may include: calculating the position difference between the horizontal positions of each pair of feature points in each matched pair of feature points, and screening out at least one pair of first feature points in each matched pair of feature points whose position difference is less than a first threshold, so as to screen out at least one pair of first image columns corresponding to the horizontal position of the at least one pair of first feature points.

[0078] Specifically, the feature extraction and screening unit 1030 may first calculate the difference between the horizontal positions (i.e., horizontal coordinate positions, in pixels) of each pair of feature points on the first panoramic image and the second panoramic image as the position difference of each pair of feature points. Subsequently, each calculated position difference may be compared with a preset first threshold, and at least one pair of first feature points whose position difference is less than the first threshold may be screened out. Figure 5 An example of filtering each pair of image columns on the first panoramic image and the second panoramic image by filtering each pair of feature points according to an embodiment of the present invention is shown. Figure 5As shown, through the above screening process, at least a pair of first feature points 501 and 502 whose position difference meets the first threshold requirement are screened out, and it can be seen that the horizontal positions of the pair of first feature points 501 and 502 are very close.

[0079] In order to further select at least one pair of first image columns corresponding to the horizontal position of the at least one pair of first feature points based on the selected at least one pair of first feature points, in one example, the feature extraction and screening unit 1030 may determine the positions of the corresponding pair of first image columns based on the average horizontal position of each pair of first feature points selected. Figure 5 As shown, the feature extraction and screening unit 1030 can calculate the average of the horizontal positions of the first feature points 501 and 502 to obtain their average horizontal positions, and screen out a pair of image columns 503 and 504 located at the average horizontal position on the first panoramic image and the second panoramic image as a pair of first image columns corresponding to the horizontal positions of the pair of first feature points 501 and 502. In addition, in another example, the feature extraction and screening unit 1030 can also screen out a corresponding pair of first image columns based on the horizontal position of one of the feature points in each pair of screened first feature points (for example, screen out a pair of image columns located at the horizontal position of the feature point 501 or the feature point 502 on the first panoramic image and the second panoramic image) as a pair of first image columns corresponding to the horizontal position of the pair of first feature points, which is not limited here.

[0080] It should be noted that there is a situation where the pre-set first threshold is too large, so that any pair of first feature points that meet the first threshold requirement cannot be screened out, and thus the first image column cannot be screened out. Therefore, when the position difference of each pair of matched feature points is greater than the first threshold, the feature extraction and screening unit 1030 can also lower the first threshold until at least one pair of first feature points whose position difference is less than the first threshold is screened out from each pair of matched feature points.

[0081] The at least one pair of first image sequences obtained through the feature point matching and screening process performed by the feature extraction and screening unit 1030 is the at least one pair of first image sequences whose feature similarity meets the predetermined requirement. Hereinafter, another scheme for the feature extraction and screening unit 1030 to screen out at least one pair of first image sequences whose feature similarity meets the predetermined requirement on the first panoramic image and the second panoramic image will be described.

[0082] Image column screening solution 2: Using convolutional neural network to obtain feature maps

[0083] In general, this solution sends the first panoramic image and the second panoramic image to a convolutional neural network (CNN) through a feature extraction and screening unit 1030 to obtain a corresponding feature map, and screens the features of each pair of image columns on the feature map to achieve screening of each pair of image columns on the first panoramic image and the second panoramic image.

[0084] In this solution, the feature extraction and screening unit 1030 may perform feature extraction processing on the first panoramic image and the second panoramic image, including: inputting the first panoramic image and the second panoramic image into a convolutional neural network to obtain at least one first feature map of the first panoramic image and at least one second feature map of the second panoramic image having the horizontal width of the panoramic image. As is well known to those skilled in the art, by inputting an image into a convolutional neural network, at least one feature map of the image may be extracted through the convolutional layer of the convolutional neural network, wherein the at least one feature map includes a plurality of feature values ​​arranged corresponding to the positions of the features on the image, and the size of the feature map may be controlled by a zero-padding operation. In this solution, the feature extraction and screening unit 1030 may obtain at least one first feature map of the first panoramic image and at least one second feature map of the second panoramic image having the same horizontal width as the first panoramic image and the second panoramic image by controlling the size of the feature map.

[0085] Figure 6 An example of implementing the filtering of each pair of image columns on the first panoramic image and the second panoramic image by filtering each pair of image column features according to an embodiment of the present invention is shown. Figure 6 , three first feature maps and three second feature maps extracted for the first panoramic image and the second panoramic image respectively are shown, but the present invention does not limit the number of the first feature maps and the second feature maps (i.e., the depth of the feature maps), nor does it limit the height of the first feature maps and the second feature maps in the vertical direction.

[0086] After obtaining the above-mentioned first feature map and second feature map, the feature extraction and screening unit 1030 can use the first feature map and the second feature map as the result of the feature extraction processing, and screen each pair of image columns with the same horizontal position on the first panoramic image and the second panoramic image based on the result of the feature extraction processing to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets predetermined requirements, including: calculating the feature difference between each pair of image column features with the same horizontal position on the first feature map and the second feature map, and screening out at least one pair of first image column features in each pair of image column features whose feature difference is less than a second threshold, so as to screen out at least one pair of first image columns corresponding to the horizontal position of the at least one pair of first image column features.

[0087] Specifically, the feature extraction and screening unit 1030 can extract image column features at each horizontal position by column (for example, from left to right or from right to left in the horizontal direction) on the first feature map and the second feature map, wherein each extracted image column feature is a feature matrix including one or more feature vectors. After extracting each image column feature, the feature extraction and screening unit 1030 can calculate the feature difference between each pair of image column features having the same horizontal position on the first feature map and the second feature map. In an example, in order to calculate the feature difference between a pair of image column features, the feature extraction and screening unit 1030 can calculate the distance (such as L2 distance, cosine similarity, etc.) between each feature vector having the same vertical position in the vertical direction from top to bottom or from bottom to top for the pair of image column features, and sum up each distance as the difference between the pair of image column features. However, it should be noted that those skilled in the art can use other means for calculating the difference between feature matrices to determine the feature difference between each pair of image column features according to the embodiment of the present invention through the feature extraction and screening unit 1030, which is not limited here.

[0088] After obtaining the feature differences between each pair of image column features having the same horizontal position on the first feature map and the second feature map, the feature extraction and screening unit 1030 can compare them with a preset second threshold value, thereby screening out at least one pair of first image column features having a feature difference less than the second threshold value. Thereafter, based on the horizontal position of the at least one pair of image column features, the feature extraction and screening unit 1030 can screen out at least one pair of image columns having the same horizontal position as the at least one pair of first image column features on the first panoramic image and the second panoramic image, as at least one pair of first image columns whose feature similarity meets a predetermined requirement. Figure 6As shown, by performing feature difference calculation and threshold comparison on each pair of image column features on the first feature map and the second feature map, the feature extraction and screening unit 1030 screens out a pair of first image column features 601 and 602 and another pair of first image column features 603 and 604 whose feature difference is less than the second threshold, and accordingly, further screens out a pair of first image columns 605 and 606 having the same horizontal position as the first image column features 601 and 602 and another pair of first image columns 607 and 608 having the same horizontal position as the first image column features 603 and 604, wherein the above two pairs of first image columns are at least one pair of first image columns screened out on the first panoramic image and the second panoramic image whose feature similarity meets predetermined requirements.

[0089] It should be noted that there is a situation where the preset second threshold is too large, so that any pair of first image sequence features that meet the second threshold requirement cannot be screened out, and thus the first image sequence cannot be screened out. Therefore, when the feature difference of each pair of image sequence features is greater than the second threshold, the feature extraction and screening unit 1030 can also lower the second threshold until at least one pair of first image sequence features whose feature difference is less than the second threshold is screened out from each pair of image sequence features.

[0090] Through the above two solutions, the feature extraction and screening unit 1030 can screen out at least one pair of first image columns whose feature similarity meets a predetermined requirement on the first panoramic image and the second panoramic image.

[0091] Subsequently, the position determining unit 1040 determines a panoramic image jump position for jumping between the first panoramic image and the second panoramic image based on the at least one pair of first image columns that are filtered out.

[0092] Generally speaking, two jump positions need to be determined on a panoramic image, namely, an exit position for jumping from the panoramic image to another panoramic image, and an arrival position for jumping from the other panoramic image to the panoramic image. In the case where the number of the at least one pair of first image columns screened out is 2 and the two pairs of first image columns happen to be located in the left half and the right half of the panoramic image, respectively, the position determination unit 1040 can directly determine the exit position on the first panoramic image and the arrival position on the second panoramic image for jumping from the first panoramic image to the second panoramic image based on the positions of the two pairs of first image columns screened out, as well as the exit position on the second panoramic image and the arrival position on the first panoramic image for jumping from the second panoramic image to the first panoramic image, thereby completing the selection of the jump positions of the panoramic images.

[0093] In addition, when the number of at least one pair of first image columns whose feature similarity meets the predetermined requirement is 1, the position determination unit 1040 can directly determine the position of another pair of first image columns by using the horizontal width of the panoramic image. Specifically, when only one pair of first image columns is selected, the position determination unit 1040 can determine another pair of first image columns at a horizontal position that is half the horizontal width of the panoramic image away from the horizontal position of the selected pair of first image columns. Figure 7 An example of determining another pair of first image columns according to the horizontal width of a panoramic image when only one pair of first image columns is screened out according to an embodiment of the present invention is shown. In which, only a pair of first image columns 701 and 702 whose feature similarity meets a predetermined requirement are screened out on the first panoramic image and the second panoramic image. At this time, the position determination unit 1040 can directly determine another pair of image columns 703 and 704 that are half the horizontal width of the panoramic image (i.e., W / 2) away from them as another pair of first image columns.

[0094] In addition, when the number of at least one pair of first image columns screened out whose feature similarity meets predetermined requirements is greater than 2, the position determination unit 1040 may perform aggregation processing on the at least one pair of first image columns screened out to aggregate them into two pairs of first image columns. Figure 8 FIG. 2 shows an example of performing aggregation processing on image columns when more than two pairs of first image columns are screened out according to an embodiment of the present invention. Figure 8 As shown on the left side of , first image columns whose feature similarities greater than two pairs meet predetermined requirements are screened out on the first panoramic image and the second panoramic image, and after aggregation processing, a pair of aggregated first image columns 801 and 802 and another pair of first image columns 803 and 804 are finally obtained. Among them, the position determination unit 1040 can implement the above-mentioned aggregation processing by aggregating the horizontal positions of at least one pair of screened out first image columns through various means, for example, clustering the horizontal positions of each first image column to find their clustering center, and taking the image column located at the clustering center as the aggregated first image column, which is not limited here. It should be noted that the position determination unit 1040 may not perform the above-mentioned aggregation processing, but directly select any two pairs of first image columns from the at least one pair of screened out first image columns, as long as the two pairs of first image columns are respectively located in the left half and the right half of the panoramic image, which is not limited here.

[0095] Fig. 9 An example of determining a departure position and an arrival position for jumping between panoramic images based on two pairs of first image sequences according to an embodiment of the present invention is shown. Fig. 9As shown, after the above processing, the position determination unit 1040 finally selects two pairs of image columns for setting jump positions on the first panoramic image and the second panoramic image, namely, a pair of first image columns 901 and 902 located in the left half of the panoramic image and a pair of first image columns 903 and 904 located in the right half of the panoramic image. Fig. 9 As further shown in FIG. 1 , the position determination unit 1040 can determine the position of the image column 901 on the first panoramic image as the arrival position for jumping from the second panoramic image to the first panoramic image, determine the position of the image column 902 on the second panoramic image as the departure position for jumping from the second panoramic image to the first panoramic image, determine the position of the image column 903 on the first panoramic image as the departure position for jumping from the first panoramic image to the second panoramic image, and determine the position of the image column 904 on the second panoramic image as the arrival position for jumping from the first panoramic image to the second panoramic image, thereby completing the setting of the jump position. In addition, a guide point can be further set at any position of the above-mentioned image column to be displayed to the user and selected by the user during the virtual roaming process.

[0096] According to the above-mentioned panoramic image jump position determination device of the present invention, the panoramic image jump position can be automatically and accurately determined so as to reasonably set the guide point, improve the efficiency of setting the guide point, and make the user's perspective remain unchanged before and after the jump, so that the user seems to jump to the next scene by actually moving forward, giving the user a better progressive roaming experience.

[0097] Below, refer to Fig.11 An image processing apparatus according to an embodiment of the present invention will be described. Fig.11 FIG. 1 is a block diagram of an image processing apparatus 1100 according to an embodiment of the present invention. Fig.11 As shown, the device 1100 may be a computer or a server.

[0098] like Fig.11 As shown, the image processing device 1100 includes one or more processors 1110 and a memory 1120. Of course, in addition to this, the image processing device 1100 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Fig.11 The components and structure of the image processing device 1100 shown are merely exemplary and not restrictive. The image processing device 1100 may also have other components and structures as required.

[0099] The processor 1110 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1120 to execute desired functions, which may include: acquiring a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in a three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; performing adjustment processing on at least one of the first panoramic image and the second panoramic image according to the posture data of the cameras for shooting the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; after the adjustment processing, performing feature extraction processing on the first panoramic image and the second panoramic image, and based on the result of the feature extraction processing, screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image, so as to screen out at least one pair of first image columns whose feature similarity meets a predetermined requirement in each pair of image columns; and determining a panoramic image jump position for jumping between the first panoramic image and the second panoramic image based on the screened at least one pair of first image columns.

[0100] The memory 1120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1110 may run the program instructions to implement the functions of the image processing device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the image processing method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0101] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions implement the following steps when executed by a processor: acquiring a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in a three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; performing adjustment processing on at least one of the first panoramic image and the second panoramic image according to posture data of cameras for shooting the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; after the adjustment processing, performing feature extraction processing on the first panoramic image and the second panoramic image, and based on the result of the feature extraction processing, screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image, so as to screen out at least one pair of first image columns in each pair of image columns whose feature similarity meets a predetermined requirement; and determining a panoramic image jump position for jumping between the first panoramic image and the second panoramic image based on the screened at least one pair of first image columns.

[0102] Of course, the above-mentioned specific embodiments are merely illustrative rather than restrictive, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.

[0103] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0104] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0105] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to a singular element, such as using the article "a", "an", or "the", is not to be construed as limiting the element to the singular.

[0106] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0107] Each operation of the above method can be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0108] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0109] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.

[0110] The method invented herein includes one or more actions for implementing the described method. The method and / or action can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.

[0111] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.

[0112] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

[0113] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.

[0114] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0115] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0116] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0117] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.

[0118] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for determining a jump position of a panoramic image, include: Acquire a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in the three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; Adjusting at least one of the first panoramic image and the second panoramic image according to the posture data of the camera that took the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; After the adjustment process, feature extraction is performed on the first panoramic image and the second panoramic image, and pairs of image columns having the same horizontal position on the first panoramic image and the second panoramic image are screened based on the result of the feature extraction process, so as to screen out at least one pair of first image columns whose feature similarity meets a predetermined requirement in each pair of image columns; as well as A panoramic image jump position for jumping between the first panoramic image and the second panoramic image is determined based on the at least one pair of filtered first image columns.

2. The method according to claim 1, wherein the first panoramic image and the second panoramic image are subjected to feature extraction processing. include: Feature point matching is performed on the first panoramic image and the second panoramic image to extract pairs of feature points matched on the first panoramic image and the second panoramic image.

3. The method of claim 2, wherein each matched pair of feature points is used as the result of the feature extraction process, and The method further comprises screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image based on the result of the feature extraction process, so as to screen out at least one pair of first image columns whose feature similarity meets a predetermined requirement in each pair of image columns. include: The position difference between the horizontal positions of each pair of feature points in each matched pair of feature points is calculated, and at least one pair of first feature points in each matched pair of feature points whose position difference is less than a first threshold is screened out to screen out at least one pair of first image columns corresponding to the horizontal position of the at least one pair of first feature points.

4. The method according to claim 1, in, The feature extraction process of the first panoramic image and the second panoramic image comprises: The first panoramic image and the second panoramic image are input into a convolutional neural network to obtain at least one first feature map of the first panoramic image and at least one second feature map of the second panoramic image having the horizontal width.

5. The method of claim 4, wherein the first feature map and the second feature map are used as the result of the feature extraction process, and The method further comprises screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image based on the result of the feature extraction process, so as to screen out at least one pair of first image columns whose feature similarity meets a predetermined requirement in each pair of image columns. include: The feature differences between each pair of image column features having the same horizontal position on the first feature map and the second feature map are calculated, and at least one pair of first image column features whose feature difference is less than a second threshold value in each pair of image column features is screened out, so as to screen out at least one pair of first image columns corresponding to the horizontal position of the at least one pair of first image column features.

6. The method according to claim 3, in, When the position differences of each matched pair of feature points are greater than a first threshold, the method further includes: increasing the first threshold until at least one pair of first feature points having a position difference less than the first threshold is screened out from each matched pair of feature points.

7. The method according to claim 5, in, When the feature difference of each pair of image column features is greater than a second threshold, the method further includes: increasing the second threshold until at least one pair of first image column features having the feature difference less than the second threshold is screened out from each pair of image column features.

8. The method according to claim 3 or 5, in, When the number of the at least one pair of first image columns screened out whose feature similarity meets the predetermined requirement is 1, the method further includes: determining another pair of first image columns at a horizontal position that is half the horizontal width away from the horizontal position of the screened pair of first image columns.

9. The method according to claim 3 or 5, in, When the number of the at least one pair of first image columns screened out whose feature similarity meets the predetermined requirement is greater than 2, the method further includes: performing aggregating processing on the at least one pair of first image columns screened out to aggregate them into two pairs of first image columns.

10. A device for determining a jump position of a panoramic image, include: an acquisition unit configured to acquire a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in a three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; an adjustment unit configured to adjust at least one of the first panoramic image and the second panoramic image according to the posture data of the camera that shoots the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; a feature extraction and screening unit, configured to perform feature extraction processing on the first panoramic image and the second panoramic image after the adjustment processing, and screen each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image based on the result of the feature extraction processing, so as to screen out at least one pair of first image columns whose feature similarity degree meets a predetermined requirement in each pair of image columns; as well as The position determination unit is configured to determine a panoramic image jump position for jumping between the first panoramic image and the second panoramic image based on the at least one pair of first image columns that are screened out.

11. A device for determining a jump position of a panoramic image, include: processor; and a memory having computer program instructions stored therein, Wherein, when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: Acquire a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in the three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; Adjusting at least one of the first panoramic image and the second panoramic image according to the posture data of the camera that took the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; After the adjustment process, performing feature extraction process on the first panoramic image and the second panoramic image, and screening each pair of image columns having the same horizontal position on the first panoramic image and the second panoramic image based on the result of the feature extraction process, so as to screen out at least one pair of first image columns whose feature similarity degree meets a predetermined requirement in each pair of image columns; and A panoramic image jump position for jumping between the first panoramic image and the second panoramic image is determined based on the at least one pair of filtered first image columns.

12. A computer-readable storage medium having computer program instructions stored thereon, in, When the computer program instructions are executed by a processor, the following steps are implemented: Acquire a first panoramic image and a second panoramic image having the same horizontal width, wherein a shooting point for shooting the first panoramic image is located in the three-dimensional space corresponding to the second panoramic image, and a shooting point for shooting the second panoramic image is located in the three-dimensional space corresponding to the first panoramic image; Adjusting at least one of the first panoramic image and the second panoramic image according to the posture data of the camera that took the first panoramic image and the second panoramic image, so as to obtain the first panoramic image and the second panoramic image corresponding to the same camera posture; After the adjustment process, feature extraction is performed on the first panoramic image and the second panoramic image, and pairs of image columns having the same horizontal position on the first panoramic image and the second panoramic image are screened based on the result of the feature extraction process, so as to screen out at least one pair of first image columns whose feature similarity meets a predetermined requirement in each pair of image columns; as well as A panoramic image jump position for jumping between the first panoramic image and the second panoramic image is determined based on the at least one pair of filtered first image columns.

Images