A panoramic image data construction method, retrieval method and system

Abstract

The application provides a panoramic image data construction method, a retrieval method and a system. The panoramic image data construction method comprises: collecting a panoramic image; obtaining camera parameters to obtain a camera model based on which the panoramic image is divided into sub-images; calculating the number of directions N when the panoramic image is divided into sub-images according to the camera parameters, N being a natural number greater than 1; projecting the panoramic image based on the camera model and according to the number of directions N to obtain N sub-images of the panoramic image; and extracting image feature data of the panoramic image sub-images. The panoramic image data comprises the panoramic image, the corresponding sub-images of the panoramic image and the image feature data. The application can effectively realize panoramic image retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer image processing, in particular to a panoramic image data construction method, retrieval method and system. BACKGROUND

[0002] With the gradual entry of panoramic cameras into ordinary consumers (GoPro, Insta360, etc.), it has become more and more convenient to obtain panoramic images. Panoramic images originate from the concept of traditional planes, and the horizontal angle of view contains a complete week, i.e. 360° view, but limited to two-dimensional space, panoramic images cannot fully present the visual effect of being in a three-dimensional space inside a sphere or a cube. Panoramic images are planar images, and when the panoramic image is unfolded into a plane, it can be seen that the farther away from the center, the greater the deformation of the panoramic image. Panoramic images are very different from the planar images taken by conventional cameras.

[0003] With the development of AR / VR technology, panoramic cameras have found another application. For example: in AR (Augmented Reality) technology, in order to more quickly and conveniently collect the surrounding environment for the generation of a visual map, a panoramic camera is often used to obtain a panoramic image; in VR (Virtual Reality) technology, virtual reality applications based on panoramic images are widespread. Panoramic images are suitable for AR / VR technology and have the advantages of convenience and low cost, and have gradually replaced polygonal mesh 3D image technology.

[0004] Image retrieval technology aims to find images containing the same instance from database images, which is a very important technology in the field of image processing. With the gradual widespread application of existing technologies to panoramic images, panoramic image retrieval technology has also become increasingly important. However, due to the different imaging mechanisms of panoramic images and conventional cameras, panoramic images cannot be directly matched with images taken by conventional cameras, and how to achieve panoramic image retrieval is a difficulty in existing image retrieval technologies. SUMMARY

[0005] The technical problem solved by the technical solution of the present application is how to more effectively achieve panoramic image retrieval.

[0006] In order to solve the above technical problems, the technical solution of the present application provides a panoramic image data construction method, comprising:

[0007] acquiring a panoramic image;

[0008] obtaining camera parameters to obtain a camera model based on which the panoramic image is segmented into sub-images;

[0009] calculating the number of directions N when segmenting the panoramic image into sub-images according to the camera parameters, N being a natural number greater than 1.

[0010] projecting the panorama image based on the camera model and according to the number of directions N to obtain N sub-images of the panorama image;

[0011] extracting image feature data of the panorama image sub-images;

[0012] The panorama image data comprises the panorama image, the sub-images corresponding to the panorama image and the image feature data.

[0013] Optionally, the collected panorama image is a 360° panorama image.

[0014] Optionally, the camera model based on which the camera parameters are obtained to divide the panorama image into sub-images comprises:

[0015] obtaining a field of view angle parameter α and a resolution parameter ω of a camera corresponding to the panorama image;

[0016] calculating camera model parameters corresponding to the division of the panorama image into sub-images based on the obtained field of view angle parameter α and the resolution parameter ω.

[0017] Optionally, the calculation of the camera model parameters corresponding to the division of the panorama image into sub-images based on the obtained field of view angle parameter α and the resolution parameter ω comprises:

[0018] obtaining a focal length parameter f of the sub-images of the panorama image under the camera model according to the field of view angle parameter α and the resolution parameter ω:

[0019]

[0020] obtaining a camera intrinsic matrix B based on the focal length f:

[0021]

[0022] wherein the resolution parameter ω is the product of the image width and the image height, f x is a value of the focal length parameter f calculated on the image width with the horizontal field of view angle, f y is a value of the focal length parameter f calculated on the image width with the vertical field of view angle, c x is a value of half of the image width, c y is a value of half of the image height;

[0023] The model parameters comprise the focal length parameter f and the intrinsic matrix B.

[0024] Optionally, the calculation of the number of directions N for dividing the panorama image into sub-images according to the camera parameters comprises determining the number of directions N for dividing the panorama image into sub-images based on the field of view angle parameter α of the camera.

[0025] Optionally, the number of directions N in splitting the panoramic image into sub-images based on the field of view angle parameter a of the camera comprises:

[0026] Supposing the field of view angle parameter of the camera is a, and the panoramic image is a 360° panoramic image, the number of directions N in splitting the panoramic image into sub-images is calculated by the following function:

[0027] N≥ceil(360° / a), wherein ceil() is a function of rounding up.

[0028] Optionally, the projecting the panoramic image based on the camera model and according to the number of directions N to obtain N sub-images of the panoramic image comprises:

[0029] Setting an initial rotation angle of the first sub-image in splitting the panoramic image;

[0030] Based on the initial rotation angle, determining N cutting directions according to the number of directions N; the interval angle between the N cutting directions is (360° / N);

[0031] Combining the initial rotation angle, the field of view angle parameter, the N cutting directions and a preset resolution parameter as a camera model;

[0032] Projecting the panoramic image to a pixel coordinate system plane based on the camera model to obtain N sub-images.

[0033] Optionally, supposing the collected panoramic image is P, and the N sub-images obtained by projecting the panoramic image P are q1, q2, q3, … qN; N The extracting the image feature data of the panoramic image sub-image comprises:

[0034] Extracting a global image feature for each sub-image q1, q2, q3, … qN; N to obtain a corresponding global image feature F1, F2, F3, … F N Each global image feature at least includes one image attribute feature extracted from the corresponding sub-image as a whole;

[0035] The panoramic image data comprises: the panoramic image P, the N sub-images q1, q2, q3, … qN, and the corresponding global image feature F1, F2, F3, … F N of each sub-image.

[0036] To solve the above technical problems, the technical scheme of the present application further provides a panoramic image data construction method, comprising:

[0037] Collecting a panoramic image p';

[0038] Obtaining a plurality of sets of camera parameters based on a plurality of sets of camera models for dividing the panoramic image p' into a plurality of sets of sub-images, the plurality of sets of camera parameters comprising: field of view angle parameters α1, α2, α3, … α z1 , resolution parameters ω1, ω2, ω3, … ω z2 ;

[0039] According to the plurality of sets of camera parameters, a plurality of sets of direction numbers n1, n2, n3, … n z3 corresponding to dividing the panoramic image p' into a plurality of sets of sub-images are calculated.

[0040] A plurality of sets of initial rotation angles θ1, θ2, θ3, … θ z4 of the first sub-image when dividing the panoramic image p' are set; z1, z2, z3, z4 are natural numbers greater than 1.

[0041] Based on the plurality of sets of camera parameters, the plurality of sets of direction numbers n1, n2, n3, … n z , and the plurality of sets of initial rotation angles, parameter combinations of camera models are performed to obtain a plurality of sets of combined camera parameters and a plurality of sets of cutting directions.

[0042] A plurality of camera models are established according to the plurality of sets of combined camera parameters.

[0043] A camera model is selected from the plurality of camera models and a corresponding cutting direction is determined, and the panoramic image P' is projected onto a pixel coordinate system plane to obtain a preset number of sub-images.

[0044] Suppose that N0 sub-images q'1, q'2, q'3, … q' N0 obtained by projecting the panoramic image P' are obtained; N0 is a preset number of sub-images obtained according to a plurality of camera models and corresponding cutting directions, and N0 is a natural number greater than 1.

[0045] Image feature data F'1, F'2, F'3, … F' N0 of the sub-images q'1, q'2, q'3, … q' N0 of the panoramic image P' are extracted.

[0046] The panoramic image data comprises: a panoramic image P', N sub-images q'1, q'2, q'3, … q' N0 , and image feature data F'1, F'2, F'3, … F' N0 corresponding to each sub-image.

[0047] In order to solve the above technical problems, the technical scheme of the present application also provides a panoramic image data retrieval method based on the panoramic image data constructed by the above method, comprising:

[0048] Receive input image;

[0049] Extract image features from the input image;

[0050] Acquire panoramic image data;

[0051] The image features of the input image are matched with the image feature data of the panoramic image data to obtain the image feature data that is closest to the image features of the input image;

[0052] Based on the image feature data obtained from the matching, the most similar panoramic image to the input image is retrieved.

[0053] Optionally, the image features are global image features; let the input image be J, and the image features extracted from the input image J be j, then the obtained panoramic image data is:

[0054] Panoramic images P1, P2, P3…P m m is a natural number greater than 1;

[0055] Panoramic images P1, P2, P3…P m The corresponding sub-image data Q1, Q2, Q3…Q m Sub-image data Q1 contains N1 panoramic images P1 and sub-images q 11 q 12 q 13 、...q 1N1 Sub-image data Q2 contains N2 panoramic images P2 and sub-images q 21 q 22 q 23 、...q 2N2 Sub-image data Q3 contains N3 panoramic images P3, and sub-image q 31 q 32 q 33 、...q 3N3 Similarly, the sub-image data Q m Contains Nm panoramic images P m sub-image q m1 q m2 q m3 、...q mNm N1, N2, N3…Nm are natural numbers greater than 1;

[0056] Sub-image data Q1, Q2, Q3…Q m The corresponding image feature data G1, G2, G3…G m Image feature data G1 contains sub-image q 11 q 12 q 13 、...q 1N1 Extracted global image features F11 , F 12 , F 13 ,... F 1N1 ; image feature data G2 contains global image features F 21 , q 22 , q 23 ,... q 2N2 extracted from sub-images q 21 , F 22 , F 23 ,... F 2N2 ; image feature data G3 contains global image features F 31 , q 32 , q 33 ,... q 3N3 extracted from sub-images q 31 , F 32 , F 33 ,... F 3N3 ; and so on, image feature data G m contains global image features F m1 , q m2 , q m3 ,... q mNm extracted from sub-images q m1 , F m2 , F m3 ,... F mNm ;

[0057] The matching of the image features of the input image with the image feature data of the panoramic image data to obtain the image feature data closest to the image features of the input image comprises:

[0058] matching the image features j of the input image J with the corresponding image feature data G1, G2, G3,... G m of the panoramic images P1, P2, P3,... P m , i.e. global image features F 11 , F 12 , F 13 ,... F 1N1 , F 21 , F 22 , F 23 ,... F 2N2 , F 31 , F 32 , F 33 ,... F 3N3 ,... F m1 , F m2 , F m3 ,... F mNm , to obtain the closest global image features F e1 , F e2 , Fe3 ey e1, e2, e3, … ey are the sequences of the sub-images corresponding to the global image features close to the distance of the image feature j;

[0059] The searching of the panoramic image most similar to the input image based on the matched image feature data comprises:

[0060] obtaining global image features F e1 e2 e3 ey corresponding matching sub-images q e1 e2 e3 ey ;

[0061] counting the number of the matching sub-images in the panoramic images P1, P2, P3…P m , and selecting the panoramic image Ps with the largest number of matching sub-images as the panoramic image most similar to the input image.

[0062] To solve the above technical problems, the technical scheme of the present application further provides a panoramic image data construction system, comprising:

[0063] a collecting unit adapted to collect panoramic images;

[0064] an obtaining unit adapted to obtain camera parameters to obtain a camera model based on which the panoramic images are segmented into sub-images;

[0065] a calculating unit adapted to calculate the number of directions N when the panoramic images are segmented into sub-images according to the camera parameters, N being a natural number greater than 1;

[0066] a projecting unit adapted to project the panoramic images based on the camera model and according to the number of directions N to obtain N sub-images of the panoramic images;

[0067] an extracting unit adapted to extract image feature data of the sub-images of the panoramic images;

[0068] a storage unit adapted to store the panoramic image data; the panoramic image data comprises the panoramic images, the sub-images corresponding to the panoramic images and the image feature data.

[0069] To solve the above technical problems, the technical scheme of the present application further provides a panoramic image data construction system, comprising:

[0070] a collecting unit adapted to collect panoramic images p’;

[0071] ​​​​​​​The acquisition unit is adapted to acquire a plurality of sets of camera parameters based on a plurality of sets of camera models for dividing the panoramic image p' into a plurality of sets of sub-images, the plurality of sets of camera parameters comprising: field of view angle parameters α1, α2, α3, … α z1 , resolution parameters ω1, ω2, ω3, … ω z2 ;

[0072] The calculation unit is adapted to calculate a plurality of sets of direction numbers n1, n2, n3, … n z3 corresponding to dividing the panoramic image p' into a plurality of sets of sub-images according to the plurality of sets of camera parameters;

[0073] The setting unit is adapted to set a plurality of sets of initial rotation angles θ1, θ2, θ3, … θ z4 of the first sub-image when dividing the panoramic image p'; z1, z2, z3, z4 are natural numbers greater than 1;

[0074] The combination unit is adapted to combine parameters of the camera model based on the plurality of sets of camera parameters, the plurality of sets of direction numbers n1, n2, n3, … n z , and the plurality of sets of initial rotation angles, to obtain a plurality of sets of combined camera parameters and a plurality of sets of cutting directions;

[0075] The establishment unit is adapted to establish a plurality of camera models according to the plurality of sets of combined camera parameters;

[0076] The projection unit is adapted to select a camera model from the plurality of camera models and determine a corresponding cutting direction, and project the panoramic image P' to a pixel coordinate system plane to obtain a preset number of sub-images; let the N0 sub-images obtained by projecting the panoramic image P' be q'1, q'2, q'3, … q' N0 ; N0 is a preset number of sub-images obtained according to the plurality of camera models and the corresponding cutting directions, and N0 is a natural number greater than 1;

[0077] The extraction unit is adapted to extract image feature data F'1, F'2, F'3, … F' N0 of the sub-images q'1, q'2, q'3, … q' N0 of the panoramic image P';

[0078] The storage unit is adapted to store the panoramic image data; the panoramic image data comprises: the panoramic image P', N sub-images q'1, q'2, q'3, … q' N0 , and image feature data F'1, F'2, F'3, … F' N0 corresponding to each sub-image.

[0079] To solve the above technical problems, the technical scheme of the present application further provides a panoramic image data retrieval system based on the panoramic image data constructed by the system as described above, comprising:

[0080] a receiving unit adapted to receive an input image;

[0081] the extraction unit is further adapted to extract image features of the input image;

[0082] the acquisition unit is further adapted to acquire panoramic image data;

[0083] a matching unit adapted to match the image features of the input image with the image feature data of the panoramic image data to obtain image feature data closest to the image features of the input image;

[0084] a retrieval unit adapted to retrieve the panoramic image most similar to the input image based on the image feature data obtained by matching.

[0085] To solve the above technical problems, the technical scheme of the present application further provides a panoramic image data construction system, comprising: a processor and a memory; the memory stores a computer program, and the processor executes the steps of the panoramic image data construction method as described above when executing the computer program.

[0086] To solve the above technical problems, the technical scheme of the present application further provides a panoramic image data retrieval system, comprising: a processor and a memory; the memory stores a computer program, and the processor executes the steps of the panoramic image data retrieval method as described above when executing the computer program.

[0087] To solve the above technical problems, the technical scheme of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program realizes the steps of the panoramic image data construction method as described above when executed by a processor.

[0088] To solve the above technical problems, the technical scheme of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program realizes the steps of the panoramic image data retrieval method as described above when executed by a processor.

[0089] The technical scheme of the present application has at least the following beneficial effects:

[0090] The technical scheme of the present application provides an image retrieval technology for panoramic image data sources, which retrieves the input image based on the multi-view projection sub-image data after the panoramic image is segmented, so as to obtain the original panoramic image corresponding to the input image. The technical scheme of the present application can effectively realize the retrieval under the panoramic image data source.

[0091] The image retrieval of the prior art aims to find images containing the same instance from the database images. Due to the different shooting angles, light, or shielding conditions of different images, and the geometric differences such as imaging deformation of panoramic images, the difficulty of image retrieval is very high. Moreover, the imaging mechanism of panoramic images is different from that of common camera images, and panoramic images cannot be directly matched with images taken by ordinary cameras. The technical scheme of the present application sets the camera parameters, pre-processes the original panoramic image, calculates the number of directions of the panoramic image according to the obtained camera parameters, projects the panoramic image from different angles to obtain the sub-images of the segmented panoramic image, further extracts the image features of these panoramic image sub-images, and thus establishes the panoramic image data for image retrieval. In the technical scheme of the present application, the retrieval of the images taken by the conventional camera can be realized based on the panoramic image data, and the panoramic image matching the taken image is obtained.

[0092] In the technical scheme of the present application, the above-mentioned camera parameters and the pre-processing method of the number of directions can be reasonably combined, and the panoramic image is segmented according to the combined parameters, so as to obtain a group of more abundant sub-images. These sub-images are closer to the images taken by different types of conventional cameras, and the geometric deformation is smaller, so that the recall rate of panoramic image retrieval can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0093] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0094] Figure 1 A flow step schematic diagram of a panoramic image data construction method provided by the technical scheme of the present application;

[0095] Figure 2 A flow step schematic diagram of another panoramic image data construction method provided by the technical scheme of the present application;

[0096] Figure 3 A flow step schematic diagram of a panoramic image data retrieval method provided by the technical scheme of the present application. DETAILED DESCRIPTION

[0097] In order to better represent the technical scheme of the present application, the present application will be further described below with reference to the drawings.

[0098] The embodiment provides a panoramic image data construction method, as shown in the following formula: Figure 1 The embodiment provides a panoramic image data construction method, as shown in the following formula:

[0099] In step S100, a panoramic image is collected.

[0100] The panoramic image can be a series of 360° panoramic images collected by a panoramic camera. The embodiment discusses the data construction process of a single panoramic image, and if multiple panoramic images are collected, the same scheme can be used to process the panoramic images in batches to form the panoramic image data constructed by the embodiment.

[0101] Suppose that the collected panoramic image is P.

[0102] In step S101, camera parameters are obtained to obtain a camera model based on which the panoramic image is segmented into sub-images.

[0103] The camera parameters can be selected according to the camera parameters in the prior art as needed to obtain the camera model required when the panoramic image is segmented into sub-images.

[0104] The camera parameters in the prior art generally refer to field of view angle parameters (FOV), pitch angle parameters (Pitch), yaw angle parameters (Yaw), resolution parameters, shooting image aspect ratio parameters and the like. According to different parameter combinations, different camera models under different combinations can be formed.

[0105] The camera parameters obtained in the embodiment can be field of view angle parameters α and resolution parameters ω.

[0106] Specifically, in step S101, the camera parameters are obtained to obtain a camera model based on which the panoramic image is segmented into sub-images, including the following process:

[0107] The field of view angle parameters α and the resolution parameters ω of the camera corresponding to the panoramic image are obtained.

[0108] The camera model parameters are calculated based on the obtained field of view angle parameters α and the resolution parameters ω.

[0109] More specifically, the camera model parameters can be obtained by referring to the following process:

[0110] According to the field of view angle parameters α and the resolution parameters ω, the focal length parameters f of the sub-images of the panoramic image under the camera model are obtained.

[0111] The camera intrinsic matrix B is obtained based on the focal length f.

[0112]

[0113] wherein the resolution parameter ω is the image width multiplied by the image height, f x is the value of the focal length parameter f calculated on the image width with the horizontal field of view angle, f y is the value of the focal length parameter f calculated on the image width with the vertical field of view angle, c x is one half of the image width, c y is one half of the image height.

[0114] The model parameters include the focal length parameter f and the intrinsic matrix B.

[0115] The following is an example according to the above step flow, assuming that the target field of view angle parameter a is 60° and the resolution parameter ω is 960x960, then the focal length parameter f under the camera model is:

[0116]

[0117] That is:

[0118] Under the resolution ω of the target sub-image being 960x960, i.e. the image width being 960 pixel values and the image height being 960 pixel values, and the field of view angle parameter a being 60°, the focal length f of the camera intrinsic is 833, the horizontal field of view angle is 60°, the calculated f is f x , the vertical field of view angle is 60°, the calculated f is f y , therefore, f x and f y are equal, both being 833; c x , c y is the center of the image, c x is one half of the image width, being 480, c y is one half of the image height, being 480. Therefore, the camera intrinsic matrix B at this time is:

[0119] Based on the camera focal length f and the intrinsic matrix B, the camera model used when the system divides the panoramic image into sub-images can be determined.

[0120] Step S102, the number of directions N when dividing the panoramic image into sub-images is calculated according to the camera parameters, N being a natural number greater than 1.

[0121] This embodiment also needs to determine the cutting direction of the camera model used when dividing the panoramic image into sub-images. Through step S102, the number of directions N of the system can be calculated based on the camera parameters, thereby determining the number of required cutting directions.

[0122] In step S102, the number N of directions for dividing the panoramic image into sub-images is determined based on the camera parameter α. Specifically, since the panoramic image is a 360° panoramic image, the number N of directions for dividing the panoramic image into sub-images can be determined based on the 360° field of view angle by the following function:

[0123] N ≥ ceil(360° / α), where ceil() is a ceiling function.

[0124] In an example based on the above process, the value of the field of view angle parameter α is 60°, and 360° / 60° is 6. Therefore, N can take a value greater than or equal to 6. For example, when N is 6, the panoramic image can be divided into 6 sub-images.

[0125] In another example based on the above process, the value of the field of view angle parameter α is 72°, and (360° / 72°) is 5. Therefore, N can take a value greater than or equal to 5. For example, when N is 6, the panoramic image can be divided into 6 sub-images.

[0126] In step S103, the panoramic image is projected based on the camera model and in the number N of directions to obtain N sub-images of the panoramic image.

[0127] The projection of the panoramic image based on the camera model and in the number N of directions to obtain N sub-images of the panoramic image includes:

[0128] An initial rotation angle θ of the first sub-image when dividing the panoramic image is set;

[0129] Based on the initial rotation angle θ, N cutting directions are determined in the number N of directions, i.e., the N cutting directions can be θ, θ+β, θ+2β, … θ+(N-1)×β; where β is the interval angle between the cutting directions, and β can be equal to (360° / N).

[0130] The initial rotation angle, the field of view angle parameter, the N cutting directions, and a preset resolution parameter are combined as a camera model.

[0131] The panoramic image is projected onto a pixel coordinate system plane based on the camera model to obtain N sub-images.

[0132] Still taking the value of the field of view angle parameter α as 60° as an example:

[0133] 360° / 60° is a value of 6. Assuming N is 6, there are 6 different cutting directions for the sub-images, meaning the panoramic image is divided into 6 sub-images. In this case, the interval angle β between the cutting directions is (360° / 6) = 60°. If the initial rotation angle θ of the first sub-image is set to 0° when segmenting the panoramic image, the cutting directions for projecting the panoramic video onto the pixel coordinate plane can be: 0°, 60°, 120°, 180°, 240°, and 300°. In this case, the field of view parameter is 60°, which is the same as the interval angle between the sub-image cutting directions, and the sub-images do not overlap, covering exactly 360°.

[0134] In another example of the above process, let the field of view parameter α be 72°, 360° / 72° be 5, and N be 6. There are 6 different cutting directions for the sub-images, thus dividing the panoramic image into 6 sub-images. In this case, the interval angle β between the cutting directions is also (360° / 6) = 60°. If the initial rotation angle θ of the first sub-image when segmenting the panoramic image is set to 0°, the cutting directions for projecting the panoramic video onto the pixel coordinate plane can be: 0°, 60°, 120°, 180°, 240°, and 300°. In this case, the field of view parameter is 72°, which is greater than the interval angle of the sub-image cutting directions. The sub-images overlap, completely covering the 360° panoramic image with redundancy. Generally speaking, a cutting method with overlapping redundancy between sub-images is more user-friendly and has a higher success rate for retrieval at any viewing angle compared to a cutting method without overlap.

[0135] Suppose that N sub-images q1, q2, q3, ... q can be obtained by projecting the panoramic image P in step S103. N .

[0136] Step S104: Extract the image feature data of the panoramic image sub-image.

[0137] After projecting the panoramic image P, N sub-images q1, q2, q3, ... q3 of the panoramic image P can be obtained. N In step S104, it is necessary to process these images q1, q2, q3, ... q N Extracting image feature data. In this embodiment, image q1, q2, q3, ... q can be extracted. N The global image features are used to obtain sub-images q1, q2, q3, ... q N Corresponding global image features F1, F2, F3…F N Each global image feature includes at least one image attribute feature extracted from the corresponding sub-image as a whole.

[0138] In the present embodiment, the corresponding global image features can be extracted from the generated panoramic image sub-images based on a deep learning-based algorithm such as the scene recognition method of NetVLAD or a traditional image recognition method such as HOG (Histogram of Oriented Gradients).

[0139] Specifically, in the present embodiment, the scene features appearing in the sub-images can be recognized by the NetVLAD scene recognition algorithm, and the corresponding geographic location information (such as coordinates), scene names, scene feature vectors, etc. can be obtained, and the above scene features are taken as the global image features of the sub-images. (Recognition: a method of extracting scene features, converting pictures into feature vectors, and taking the similarity of feature vectors in high-dimensional space as the similarity of pictures.)

[0140] In other embodiments, the HOG (Histogram of Oriented Gradients) feature extraction method can also be used to describe the features of the objects in the sub-images, and the global image features are formed by calculating and counting the gradient direction histogram of the local region of the sub-image.

[0141] More specifically, since each sub-image contains the overall graphical attribute features such as color features, texture features, and shape features, the global image features mainly refer to the overall graphical attribute features of the scene pictures. In the extraction of global features, at least for each group of panoramic image data sources, the types of global image features extracted from the sub-images of these panoramic images are consistent, for example: if the global features extracted from the panoramic image P1 are color features, then the global image features extracted from the panoramic images P2, P3…P m are also color features; if the global image features extracted from the panoramic image P1 are texture features, then the global image features extracted from the panoramic images P2, P3…P m are also texture features; if the global features extracted from the panoramic image P1 are shape features, then the global image features extracted from the panoramic images P2, P3…P mThe extracted global features are also shape features. Multiple global features can be extracted for each panoramic image, but the extraction methods and types for sub-images from panoramic images of the same data source are consistent. In specific applications, for extracting image color features, methods such as color histogram extraction, dominant hue histogram extraction, color moment extraction, or color set extraction can be used, extracting color histogram information, dominant hue histogram information, color moment information, or color set information as color features. For extracting image texture features, methods such as LBP (Local Binary Patterns) or gray-level co-occurrence matrix extraction can be used. For extracting image shape features, edge feature extraction methods can be used, specifically implemented using Canny operator edge detection or Sobel operator edge detection algorithms.

[0142] In the technical solution of this invention, the global feature extraction of sub-images can also employ other existing image feature extraction algorithms. This invention does not impose any limitations on the proposed solution.

[0143] Step S105: Store the panoramic image data, which includes: a panoramic image, N sub-images, and global image features corresponding to each sub-image.

[0144] Based on the above steps S100 to S104, the panoramic image data obtained includes: a panoramic image P, and N sub-images q1, q2, q3, ... q N And the global image features F1, F2, F3…F corresponding to each sub-image N Storing these panoramic image data in a preset manner can construct a panoramic image data source for panoramic image retrieval, as per the technical solution of this invention.

[0145] Based on the panoramic image data construction method of this embodiment, when multiple panoramic images are acquired, i.e., panoramic images P1, P2, P3...P... are acquired... m Let m be a natural number greater than 1, then:

[0146] After steps S101 to S103, the panoramic images P1, P2, P3...P m By projecting and segmenting, we can obtain panoramic images P1, P2, P3…P m The corresponding sub-image data Q1, Q2, Q3…Q m Wherein: Sub-image data Q1 contains N1 panoramic images P1, and sub-images q 11 q 12 q 13 、...q 1N1 Sub-image data Q2 contains N2 panoramic images P2 and sub-images q21 q 22 q 23 、...q 2N2 Sub-image data Q3 contains N3 panoramic images P3, and sub-image q 31 q 32 q 33 、...q 3N3 Similarly, the sub-image data Q m Contains Nm panoramic images P m sub-image q m1 q m2 q m3 、...q mNm N1, N2, N3...Nm are natural numbers greater than 1.

[0147] Step S104 processes the sub-image data Q1, Q2, Q3…Q m Image feature extraction is performed on the sub-images in the dataset to obtain sub-image data Q1, Q2, Q3…Q. m The corresponding image feature data G1, G2, G3…G m Wherein: Image feature data G1 contains sub-image q 11 q 12 q 13 、...q 1N1 Extracted global image features F 11 F 12 F 13 ...F 1N1 Image feature data G2 contains sub-image q 21 q 22 q 23 、...q 2N2 Extracted global image features F 21 F 22 F 23 ...F 2N2 Image feature data G3 contains sub-image q 31 q 32 q 33 、...q 3N3 Extracted global image features F 31 F 32 F 33 ...F 3N3 Similarly, the image feature data G m Including sub-image q m1 q m2 q m3 、...q mNm Extracted global image features F m1 F m2 F m3 ...FmNm .

[0148] Having acquired multiple panoramic images as described above, the panoramic image data constructed based on steps S100 to S105 includes: panoramic images P1, P2, P3…P m Sub-image data Q1, Q2, Q3…Q m ; and image feature data G1, G2, G3…G m .

[0149] Corresponding to the above-described method for constructing panoramic image data, this embodiment also provides a system for constructing panoramic image data, including:

[0150] The acquisition unit is adapted to perform step S100 to acquire panoramic images;

[0151] The acquisition unit is adapted to perform step S101 to acquire camera parameters to obtain the camera model on which the panoramic image is segmented into sub-images;

[0152] The calculation unit is adapted to perform step S102 to calculate the number of directions N when segmenting the panoramic image into sub-images based on the camera parameters, where N is a natural number greater than 1.

[0153] The projection unit is adapted to perform step S103 to project the panoramic image based on the camera model and in the number of directions N to obtain N sub-images of the panoramic image.

[0154] The extraction unit is adapted to perform step S104 to extract image feature data of the panoramic image sub-image;

[0155] The storage unit is adapted to perform step S105 to store the panoramic image data.

[0156] Unlike the above embodiments, such as Figure 2 The method for constructing panoramic image data, as shown, is based on multiple sets of camera parameter models. It calculates the corresponding number of directions and determines the initial rotation angles when projecting multiple sets of panoramic images. These parameters are combined to establish multiple sets of camera models and corresponding cutting directions. The panoramic image is then projected from these multiple sets of camera models and corresponding cutting directions to form multiple sub-images, resulting in richer panoramic image sub-images. The specific steps include:

[0157] Step S200: Acquire panoramic image p'.

[0158] Step S200 is the same as step S100. A panoramic camera can be used to capture a panoramic image. In order to distinguish it from the panoramic image in step S100, the panoramic image in this embodiment is denoted as P'.

[0159] Step S201, obtaining a plurality of sets of camera parameters based on which the panoramic image p' is segmented into a plurality of sets of sub-images, the plurality of sets of camera parameters including: field of view angle parameters a1, a2, a3, … a z1 , resolution parameters w1, w2, w3, … w z2 .

[0160] In order to improve the recall rate of image retrieval, the construction method of panoramic image data is further refined in this embodiment, and a plurality of camera parameters are set. Therefore, unlike step S101, a plurality of sets of field of view angle parameters a1, a2, a3, … a z1 , resolution parameters w1, w2, w3, … w z2 are set in step S201. Further, the following parameters can be set:

[0161] A plurality of field of view angle parameters are set, such as 60°, 72°, and 90°.

[0162] A plurality of resolution parameters are set, such as 640x640, 960x960, 960x720, 720x960, etc. These resolution parameters also correspond to a plurality of aspect ratios: resolution 640x640 corresponds to an aspect ratio of 1:1, resolution 960x960 corresponds to an aspect ratio of 1:1, resolution 960x720 corresponds to an aspect ratio of 4:3, and resolution 720x960 corresponds to an aspect ratio of 3:4.

[0163] Step S202, according to the plurality of sets of camera parameters, a plurality of sets of direction numbers n1, n2, n3, … n z3 are calculated when the panoramic image p' is segmented into a plurality of sets of sub-images.

[0164] Based on the calculation method of the direction number N based on the camera parameters in step S102. Similarly, in step S202, according to the plurality of sets of field of view angle parameters a1, a2, a3, … a z1 , a plurality of sets of direction numbers n1, n2, n3, … n z3 are calculated based on the calculation method of step S102.

[0165] For example, corresponding to the plurality of field of view angle parameters 60°, 72°, and 90°, based on the formula:

[0166] N≥ceil(360° / a), and by substituting the field of view angle parameters 60°, 72°, and 90°, the corresponding direction numbers 6, 8, and 4 can be calculated. When selecting the direction number, according to the field of view angle parameter, only the direction number N x a≥360° needs to be satisfied.

[0167] Step S203, set multiple sets of initial rotation angles θ1, θ2, θ3, … θ of the first sub-image when dividing the panoramic image p'. z4 .

[0168] In step S203, the system can set multiple initial angles of different cutting directions, such as 0°, 45°, etc., to obtain panoramic image sub-images with smaller geometric deformation.

[0169] In this embodiment, the z1, z2, z3, z4 under the parameter symbol subscript are natural numbers greater than 1, representing the number of types of each parameter combination.

[0170] Step S204, based on the multiple sets of camera parameters, multiple sets of direction numbers n1, n2, n3, … n, multiple sets of initial rotation angles θ1, θ2, θ3, … θ z3 , multiple sets of initial rotation angles θ1, θ2, θ3, … θ z4 , the parameter combination of the camera model is carried out to obtain multiple sets of combined camera parameters and multiple sets of cutting directions.

[0171] The preprocessing method indicated by steps S201 to S204 in this embodiment forms a reasonable combination of data parameters, and the panoramic image is divided according to the combined parameters, which can obtain a group of more abundant sub-images. These sub-images will be closer to the images taken by different types of conventional cameras, with smaller geometric deformation, so that the recall rate of image retrieval can be improved.

[0172] Based on the processing scheme of multiple sets of data parameters in this embodiment, a series of typical parameter combinations can be obtained. The first parameter combination is: the field of view angle is 90°, the resolution is 960×960, the direction number is 4, and the initial rotation angle is 45°. The second parameter combination is: the field of view angle is 60°, the resolution is 960×960, the direction number is 6, and the initial rotation angle is 0°. The third parameter combination is: the field of view angle is 72°, the resolution is 640×640, the direction number is 8, and the initial rotation angle is 0°. The fourth parameter combination is: the field of view angle is 90°, the resolution is 960×720 or 720×960, the direction number is 8, and the initial rotation angle is 45°.

[0173] Step S205, according to the multiple sets of combined camera parameters, multiple camera models are established.

[0174] According to different parameter combinations, the system can obtain different camera models for the panoramic image to project and divide into different sub-images.

[0175] Step S206, selecting a camera model from the multiple camera models and determining the corresponding cutting direction, projecting the panoramic image P' to the pixel coordinate system plane to obtain a preset number of sub-images.

[0176] Different from step S103, based on steps S204 to S206 of the present embodiment, the panoramic image P' is projected through different camera models and corresponding cutting directions, and sub-images with more rich features can be obtained, so that the distortion of the panoramic image data is smaller compared with the image captured by a conventional camera, and the recall rate of image retrieval can be improved.

[0177] Taking the four typical camera parameter combinations in step S204 as examples, that is:

[0178] The first parameter combination is that the field of view angle a1 is 90°, the resolution w1 is 960x960, the number of directions n1 is 4, the initial rotation angle q1 is oblique 45°, and the interval angle b1 between the cutting directions is (360° / 4) = 90°. The second parameter combination is that the field of view angle a2 is 60°, the resolution w2 is 960x960, the number of directions n2 is 6, the initial rotation angle q2 is forward 0°, and the interval angle b2 between the cutting directions is (360° / 6) = 60°. The third parameter combination is that the field of view angle a3 is 72°, the resolution w3 is 640x640, the number of directions n3 is 8, the initial rotation angle q3 is forward 0°, and the interval angle b3 between the cutting directions is (360° / 8) = 45°. The fourth parameter combination is that the field of view angle a3 is 90°, the resolution w4 is 960x720, the number of directions n4 is 8, the initial rotation angle q4 is oblique 45°, and the interval angle b4 between the cutting directions is (360° / 8) = 45°.

[0179] Through the first parameter combination, the camera model and the number of directions can be obtained, and the corresponding cutting directions are: -45°, 45°, 135°, 225°. Based on the camera model and the corresponding cutting directions, the panoramic image P' is projected onto the pixel coordinate system plane to obtain four sub-images.

[0180] Through the second parameter combination, the camera model and the number of directions can be obtained, and the corresponding cutting directions are: 0°, 60°, 120°, 180°, 240°, 300°. Based on the camera model and the corresponding cutting directions, the panoramic image P' is projected onto the pixel coordinate system plane to obtain six sub-images.

[0181] Through the third parameter combination, the camera model and the number of directions can be obtained, and the corresponding cutting directions are: 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°. Based on the camera model and the corresponding cutting directions, the panoramic image P' is projected onto the pixel coordinate system plane to obtain eight sub-images.

[0182] The camera model and the number of directions are obtained by the fourth parameter combination, which can correspond to the cutting directions: -45°, 0°, 45°, 90°, 135°, 180°, 225°, and 270°. The panoramic image P' is projected onto the pixel coordinate system plane based on the camera model and the corresponding cutting direction to obtain eight sub-images.

[0183] The panoramic image P' is projected by the above four different camera models and the corresponding cutting directions, and finally 26 sub-images (i.e., the sum of the above four sub-images, six sub-images, eight sub-images, and eight sub-images) are obtained.

[0184] Based on step S206, the panoramic image P' is projected, and thus N0 sub-images can be obtained, i.e., sub-images q'1, q'2, q'3, … q' N0 . Wherein N0 is a preset number of sub-images obtained according to a plurality of camera models and corresponding cutting directions, which is also the total number of sub-images obtained based on the panoramic image P' through a plurality of camera models and corresponding cutting directions. N0 is a natural number greater than 1. N0 is the sum of the values of n1, n2, n3, … n z3 .

[0185] Step S207, extracting image feature data of the sub-images of the panoramic image P'.

[0186] Step S207 is to extract the image feature data F'1, F'2, F'3, … F' N0 of the sub-images q'1, q'2, q'3, … q' N0 of the panoramic image P'.

[0187] The method process of step S207 for extracting image feature data of the panoramic image sub-images can refer to step S104, which will not be described here.

[0188] Step S208, storing the panoramic image data, wherein the panoramic image data includes: the panoramic image P', N0 sub-images, and image feature data corresponding to each sub-image.

[0189] Specifically, the panoramic image data includes: the panoramic image P', N0 sub-images q'1, q'2, q'3, … q' N0 , and image feature data F'1, F'2, F'3, … F' N0 corresponding to each sub-image.

[0190] Based on the above panoramic image data construction method, the embodiment also provides a panoramic image data construction system, which includes:

[0191] The acquisition unit is adapted to perform step S200 to acquire the panoramic image P';

[0192] The acquisition unit is adapted to perform step S201 to acquire a plurality of sets of camera parameters based on which a plurality of sets of camera models are obtained when the panoramic image p' is segmented into a plurality of sets of sub-images, the plurality of sets of camera parameters comprising: field of view angle parameters α1, α2, α3, … α z1 , resolution parameters ω1, ω2, ω3, … ω z2 ;

[0193] The calculation unit is adapted to perform step S202 to calculate a plurality of sets of direction numbers n1, n2, n3, … n z3 corresponding to segmentation of the panoramic image p' into a plurality of sets of sub-images according to the plurality of sets of camera parameters;

[0194] The setting unit is adapted to perform step S203 to set a plurality of sets of initial rotation angles θ1, θ2, θ3, … θ z4 of the first sub-image when the panoramic image p' is segmented; z1, z2, z3, z4 are each a natural number greater than 1;

[0195] The combination unit is adapted to perform step S204 to perform parameter combination of camera models based on the plurality of sets of camera parameters, the plurality of sets of direction numbers n1, n2, n3, … n z , and the plurality of sets of initial rotation angles to obtain a plurality of sets of combined camera parameters and a plurality of sets of cutting directions;

[0196] The establishment unit is adapted to perform step S205 to establish a plurality of camera models according to the plurality of sets of combined camera parameters;

[0197] The projection unit is adapted to perform step S206 to select a camera model from the plurality of camera models and determine a corresponding cutting direction, project the panoramic image P' to a pixel coordinate system plane to obtain a preset number of sub-images; let the N0 sub-images obtained by projecting the panoramic image P' be q'1, q'2, q'3, … q' N0 ; N0 is a preset number of sub-images obtained according to the plurality of camera models and the corresponding cutting direction, and N0 is a natural number greater than 1;

[0198] The extraction unit is adapted to perform step S207 to extract image feature data F'1, F'2, F'3, … F' N0 of the sub-images q'1, q'2, q'3, … q' N0 of the panoramic image P';

[0199] The storage unit is adapted to perform step S208 to store the panoramic image data; the panoramic image data comprises: the panoramic image P', the N sub-images q'1, q'2, q'3, … q' N0 , and the image feature data F'1, F'2, F'3, … F' corresponding to each sub-image.N0 .

[0200] Based on the construction method of panoramic image data described above, the embodiment based on the panoramic image data formed by the construction method further provides a panoramic image data retrieval method as shown in Figure 3 , which can realize panoramic image retrieval and includes the following steps.

[0201] Step S300, receiving an input image.

[0202] The input image is an image taken by a conventional camera. The system receives the input image for image retrieval.

[0203] Step S301, extracting image features of the input image.

[0204] The extraction of the image features of the input image can refer to the extraction method of step S104, which will not be described here again.

[0205] In the embodiment, the extracted image features are global image features. It is assumed that the input image of step S300 is J, and the image features of the input image J extracted through step S301 are j.

[0206] Step S302, obtaining panoramic image data.

[0207] The panoramic image data obtained in step S302 can be global image data formed based on any method in the construction method of panoramic image data described above.

[0208] In the embodiment, it is assumed that the obtained panoramic image data is based on a plurality of panoramic image data sources, i.e., panoramic images P1, P2, P3…P m , and the panoramic image data can be:

[0209] The panoramic images P1, P2, P3…P m , and m is a natural number greater than 1;

[0210] The panoramic images P1, P2, P3…P m , the corresponding sub-image data Q1, Q2, Q3…Q m ; the sub-image data Q1 contains N1 sub-images q 11 , q 12 , q 13 , … q 1N1 of the panoramic image P1; the sub-image data Q2 contains N2 sub-images q 21 , q 22 , q 23 , … q 2N2 of the panoramic image P2; and the sub-image data Q3 contains N3 sub-images q 31 , q 32 , q33 , … q 3N3 ; and so on, the sub-image data Q m contains Nm panoramic images P m sub-images q m1 , q m2 , q m3 , … q mNm ; N1, N2, N3…Nm are natural numbers greater than 1;

[0211] sub-image data Q1, Q2, Q3…Q m corresponding image feature data G1, G2, G3…G m ; the image feature data G1 contains global image features F 11 , F 12 , F 13 , … F 1N1 extracted from the sub-images q 11 , q 12 , q 13 , … q 1N1 ; the image feature data G2 contains global image features F 21 , F 22 , F 23 , … F 2N2 extracted from the sub-images q 21 , q 22 , q 23 , … q 2N2 ; the image feature data G3 contains global image features F 31 , F 32 , F 33 , … F 3N3 extracted from the sub-images q 31 , q 32 , q 33 , … q 3N3 ; and so on, the image feature data G m contains global image features F m1 , F m2 , F m3 , … F mNm extracted from the sub-images q m1 , q m2 , q m3 , … q mNm .

[0212] Step S303, matching the image features of the input image with the image feature data of the panoramic image data to obtain the image feature data closest to the image features of the input image.

[0213] In step S303, the matching of the image feature of the input image with the image feature data of the panoramic image data to obtain the image feature data closest to the image feature of the input image comprises the following method process:

[0214] The image feature j of the input image J is matched with the corresponding image feature data G1, G2, G3…G m of the panoramic image P1, P2, P3…P m , i.e. global image features F 11 , F 12 , F 13 , … F 1N1 , F 21 , F 22 , F 23 , … F 2N2 , F 31 , F 32 , F 33 , … F 3N3 , … F m1 , F m2 , F m3 , … F mNm , to obtain the global image feature F e1 , F e2 , F e3 , … F ey closest to the image feature j, and e1, e2, e3, … ey are the sequences of the sub-images corresponding to the global image features closest to the image feature j.

[0215] The global image features F e1 , F e2 , F e3 , … F ey are the image feature data closest to the image feature j of the input image J.

[0216] More specifically, the distance between the image feature j and the global image features F 11 , F 12 , F 13 , … F 1N1 , F 21 , F 22 , F 23 , … F 2N2 , F 31 , F 32 , F 33 , … F 3N3 , … F m1 , F m2 , F m3 , … F mNm may be compared using the direct comparison method, i.e. j and the global image features F 11, F 12 , F 13 , … F 1N1 , F 21 , F 22 , F 23 , … F 2N2 , F 31 , F 32 , F 33 , … F 3N3 , … F m1 , F m2 , F m3 , … F mNm between the global features F e1 , F e2 … F ey , and the image features j of the input image J, so as to set the similarity between the matched global features by the threshold of the distance, so as to quickly match a plurality of global features F e1 , F e2 … F ey that meet the predetermined distance (threshold). In addition, the KNN algorithm (K-Nearest Neighbor) can also be used to select a plurality of global features F e1 , F e2 … F ey that meet the threshold.

[0217] Step S304, retrieving the panoramic image most similar to the input image based on the image feature data obtained by matching.

[0218] In step S304, the retrieving the panoramic image most similar to the input image based on the image feature data obtained by matching comprises:

[0219] obtaining the corresponding matching sub-image q e1 , q e2 , q e3 , … q ey of the global image feature F e1 , F e2 , F e3 , … F ey ;

[0220] counting the number of panoramic images P1, P2, P3… P m containing the corresponding matching sub-image, for example, the matching sub-image q e1 , q e2 , q e3 , … q eyThe number of sub-images corresponding to the panoramic image P1 is r1, the number of sub-images corresponding to the panoramic image P2 is r2, the number of sub-images corresponding to the panoramic image P3 is r3, and so on, and the number of sub-images corresponding to the panoramic image P m .

[0221] The panoramic image Ps with the largest number of matching sub-images is selected as the panoramic image most similar to the input image.

[0222] For example, in the panoramic images P1, P2, P3, … P m , the number of matching sub-images is r1, r2, r3, … rm, respectively, and the largest value among r1, r2, r3, … rm is rx, then the panoramic image corresponding to rx is the panoramic image Ps with the largest number of matching sub-images.

[0223] Corresponding to the retrieval method of the panoramic image data, based on the panoramic image data constructed by the construction system (or the construction method), the embodiment further provides a retrieval system of panoramic image data, comprising:

[0224] The receiving unit is adapted to perform step S300 to receive the input image;

[0225] The extraction unit is further adapted to perform step S301 to extract the image features of the input image;

[0226] The acquisition unit is further adapted to perform step S302 to acquire the panoramic image data;

[0227] The matching unit is adapted to perform step S303 to match the image features of the input image with the image feature data of the panoramic image data to obtain the image feature data closest to the image features of the input image;

[0228] The retrieval unit is adapted to perform step S304 to retrieve the panoramic image most similar to the input image based on the image feature data obtained by matching.

[0229] Based on the above-mentioned method and system of the embodiment, the embodiment further provides a construction system of panoramic image data, comprising a processor and a memory; the memory stores a computer program, and the processor executes the steps of the construction method of the panoramic image data as described in steps S100 to S105 when executing the computer program, or the processor can also execute the steps of the construction method of the panoramic image data as described in steps S200 to S208 when executing the computer program in other embodiments.

[0230] Based on the above method and system of the embodiment, the embodiment further provides a panoramic image data retrieval system, comprising a processor and a memory; the memory stores a computer program, and the processor executes the computer program to perform the steps of the panoramic image data retrieval method as described in steps S300 to S304.

[0231] Based on the above method and system of the embodiment, the embodiment further provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to perform the steps of the panoramic image data construction method as described in steps S100 to S105, or the computer program is also executed by the processor to perform the steps of the panoramic image data construction method as described in steps S200 to S208.

[0232] Based on the above method and system of the embodiment, the embodiment further provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to perform the steps of the panoramic image data retrieval method as described in steps S300 to S304.

[0233] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application.

Claims

1. A method for constructing panoramic image data, characterized in that, include: Acquire panoramic images; Obtain camera parameters to obtain the camera model on which the panoramic image is segmented into sub-images; The camera model used to obtain camera parameters to segment the panoramic image into sub-images includes: Obtain the field of view parameter α and resolution parameter ω of the camera corresponding to the panoramic image; Based on the acquired field of view parameter α and resolution parameter ω, calculate the multiple sets of camera model parameters corresponding to the segmentation of the panoramic image into sub-images, assuming the field of view parameter α is α1, α2, α3, ... α z1 Let the resolution parameter ω be ω1, ω2, ω3, ... ω z2 The calculation of multiple sets of camera model parameters corresponding to the segmentation of the panoramic image into sub-images based on the acquired field of view parameter α and resolution parameter ω includes: Based on the field of view parameter α and the resolution parameter ω, calculate the focal length parameter f of the sub-image of the panoramic image in the camera model: ； Based on the focal length f, the camera is obtained. Intrinsic parameter matrix B : Wherein, the resolution parameter ω is the image width multiplied by the image height, f x f is the value of the focal length parameter f calculated over the image width in terms of the lateral field of view. y c is the value calculated from the vertical field of view over the image width, where f is the focal length parameter. x c is half the width of the image. y It is half the image height; The model parameters include: focal length parameter f and intrinsic parameter matrix B; based on the camera focal length f and intrinsic parameter matrix B, the system determines multiple sets of camera models used when segmenting a panoramic image into sub-images. The number of directions N for segmenting the panoramic image into sub-images is calculated based on the multiple sets of camera parameters, where N is a natural number greater than 1. The calculation of the number of directions N for segmenting the panoramic image into sub-images based on the multiple sets of camera parameters includes: determining the number of directions N for segmenting the panoramic image into sub-images based on the camera's field of view parameter α; the determination of the number of directions N for segmenting the panoramic image into sub-images based on the camera's field of view parameter α includes: Let the field of view parameter of the camera be α, and the panoramic image be a 360° panoramic image. Then, the number of directions N when segmenting the panoramic image into sub-images is calculated using the following function: N≥ceil(360° / α), where ceil() is a function for rounding up; based on the multiple camera models and projecting the panoramic image according to the number N directions to obtain N sub-images of the panoramic image; the process of projecting the panoramic image according to the camera models and the number N directions to obtain N sub-images of the panoramic image includes: Set the initial rotation angle θ of the first sub-image when segmenting the panoramic image; Based on the initial rotation angle θ, N cutting directions are determined according to the number of directions N; the interval angle β between the N cutting directions is (360° / N); the N cutting directions are: θ, θ+β, θ+2β, ... θ+(N-1)×β; The camera model is formed by combining the initial rotation angle, field of view parameters, N cutting directions, and preset resolution parameters. Based on the camera model, the panoramic image is projected onto the pixel coordinate plane to obtain N sub-images; Extract image feature data from the sub-images of the panoramic image; The panoramic image data includes: the panoramic image, the sub-images corresponding to the panoramic image, and image feature data.

2. The method for constructing panoramic image data as described in claim 1, characterized in that, The captured panoramic images are 360° panoramic images.

3. The method for constructing panoramic image data as described in claim 1 or 2, characterized in that, ... The acquired panoramic image is P, and the N sub-images obtained by projecting the panoramic image P are q1, q2, q3, ... q N The extraction of image feature data from the panoramic sub-images includes: For each sub-image q1, q2, q3, ... q N Extract global image features to obtain corresponding global image features F1, F2, F3…F N Each global image feature includes at least one image attribute feature extracted from the corresponding sub-image as a whole; The panoramic image data includes: a panoramic image P, and N sub-images q1, q2, q3, ... q N And the global image features F1, F2, F3…F corresponding to each sub-image N .

4. A method for constructing panoramic image data, based on the method for constructing panoramic image data as described in any one of claims 1 to 3, characterized in that, Also includes: Acquire panoramic image p'; The method for obtaining multiple sets of camera parameters to segment a panoramic image p' into multiple sub-images is to use a multi-set camera model. These multiple sets of camera parameters include: field-of-view parameters α1, α2, α3, ... α z1 Resolution parameters ω1, ω2, ω3, ... ω z2 ; The number of directions n1, n2, n3, ... n corresponding to segmenting the panoramic image p' into multiple sub-images is calculated based on the multiple sets of camera parameters. z3 ; When segmenting the panoramic image p', set multiple initial rotation angles θ1, θ2, θ3, ... θ for the first sub-image. z4 z1, z2, z3, and z4 are natural numbers greater than 1. Based on the multiple sets of camera parameters and the multiple sets of direction quantities n1, n2, n3, ... n z Multiple initial rotation angles are used to combine the parameters of the camera model to obtain multiple sets of combined camera parameters and multiple sets of cutting directions; Multiple camera models are established based on the combined camera parameters. Select a camera model from the multiple camera models and determine the corresponding cutting direction, then project the panoramic image P' onto the pixel coordinate system plane to obtain a preset number of sub-images; Let N0 sub-images obtained by projecting the panoramic image P' be q'1, q'2, q'3, ..., q' N0 N0 is the preset number of sub-images obtained based on multiple camera models and corresponding cutting directions; N0 is a natural number greater than 1. Extract the sub-images q'1, q'2, q'3, ..., q' from the panoramic image P'. N0 Image feature data F'1, F'2, F'3…F' N0 ; The panoramic image data includes: a panoramic image P', and N sub-images q'1, q'2, q'3, ... q' N0 And the image feature data F'1, F'2, F'3…F' corresponding to each sub-image. N0 .

5. A method for retrieving panoramic image data, based on panoramic image data constructed according to any one of claims 1 to 4, characterized in that, include: Receive input image; Extract image features from the input image; Acquire panoramic image data; The image features of the input image are matched with the image feature data of the panoramic image data to obtain the image feature data that is closest to the image features of the input image; Based on the image feature data obtained from the matching, the most similar panoramic image to the input image is retrieved.

6. The panoramic image data retrieval method as described in claim 5, characterized in that, The image features are global image features; let the input image be J, and the image features extracted from the input image J be j, then the obtained panoramic image data is: Panoramic images P1, P2, P3…P m m is a natural number greater than 1; Panoramic images P1, P2, P3…P m The corresponding sub-image data Q1, Q2, Q3…Q m Sub-image data Q1 contains N1 panoramic images P1 and sub-images q 11 q 12 q 13 、...q 1N1 Sub-image data Q2 contains N2 panoramic images P2 and sub-images q 21 q 22 q 23 、...q 2N2 Sub-image data Q3 contains N3 panoramic images P3, and sub-image q 31 q 32 q 33 、...q 3N3 Similarly, the sub-image data Q m Contains Nm panoramic images P m sub-image q m1 q m2 q m3 、...q mNm N1, N2, N3…Nm are natural numbers greater than 1; Sub-image data Q1, Q2, Q3…Q m The corresponding image feature data G1, G2, G3…G m Image feature data G1 contains sub-image q 11 q 12 q 13 、...q 1N1 Extracted global image features F 11 F 12 F 13 ...F 1N1 Image feature data G2 contains sub-image q 21 q 22 q 23 、...q 2N2 Extracted global image features F 21 F 22 F 23 ...F 2N2 Image feature data G3 contains sub-image q 31 q 32 q 33 、...q 3N3 Extracted global image features F 31 F 32 F 33 ...F 3N3 Similarly, the image feature data G m Including sub-image q m1 q m2 q m3 、...q mNm Extracted global image features F m1 F m2 F m3 ...F mNm ; The step of matching the image features of the input image with the image feature data of the panoramic image data to obtain the image feature data that is closest to the image features of the input image includes: The image features j of the input image J are compared with the panoramic images P1, P2, P3...P m The corresponding image feature data G1, G2, G3…G m That is, global image features F 11 F 12 F 13 ...F 1N1 F 21 F 22 F 23 ...F 2N2 F 31 F 32 F 33 ...F 3N3 , ...F m1 F m2 F m3 ...F mNm Matching is performed to obtain the nearest global image feature F. e1 F e2 F e3 ...F ey e1, e2, e3, ..., ey are sequences of sub-images corresponding to global image features that are close to the image feature j; The panoramic image most similar to the input image retrieved based on the matched image feature data includes: Obtain global image features F e1 F e2 F e3 ...F ey The corresponding matching sub-image q e1 q e2 q e3 、...q ey ; Statistical analysis of panoramic images P1, P2, P3…P m The number of matching sub-images is used to select the panoramic image Ps with the most matching sub-images as the panoramic image most similar to the input image.

7. A system for constructing panoramic image data, characterized in that, include: Acquisition unit, suitable for acquiring panoramic images; The acquisition unit is adapted to acquire camera parameters to obtain the camera model on which the panoramic image is segmented into sub-images; The camera model used to obtain camera parameters to segment the panoramic image into sub-images includes: Obtain the field of view parameter α and resolution parameter ω of the camera corresponding to the panoramic image; Based on the acquired field of view parameter α and resolution parameter ω, multiple sets of camera model parameters corresponding to the segmentation of the panoramic image into sub-images are calculated; the calculation of multiple sets of camera model parameters corresponding to the segmentation of the panoramic image into sub-images based on the acquired field of view parameter α and resolution parameter ω includes: Based on the field of view parameter α and the resolution parameter ω, calculate the focal length parameter f of the sub-image of the panoramic image in the camera model: ； Based on the focal length f, the camera is obtained. Intrinsic parameter matrix B : Wherein, the resolution parameter ω is the image width multiplied by the image height, f x f is the value of the focal length parameter f calculated over the image width in terms of the lateral field of view. y c is the value calculated from the vertical field of view over the image width, where f is the focal length parameter. x c is half the width of the image. y It is half the image height; The model parameters include: focal length parameter f and intrinsic parameter matrix B; based on the camera focal length f and intrinsic parameter matrix B, the system determines multiple sets of camera models used when segmenting a panoramic image into sub-images. The calculation unit is adapted to calculate the number of multiple directions N for segmenting the panoramic image into sub-images based on the multiple sets of camera parameters, where N is a natural number greater than 1; the calculation of the number of directions N for segmenting the panoramic image into sub-images based on the camera parameters includes: determining the number of directions N for segmenting the panoramic image into sub-images based on the camera's field of view parameter α; the determination of the number of directions N for segmenting the panoramic image into sub-images based on the camera's field of view parameter α includes: Let the field of view parameter of the camera be α, and the panoramic image be a 360° panoramic image. Then, the number of directions N when segmenting the panoramic image into sub-images is calculated using the following function: N≥ceil(360° / α), where ceil() is a function that rounds up; A projection unit is adapted to project the panoramic image based on the multiple sets of camera models and in the number N directions to obtain N sub-images of the panoramic image; the projection of the panoramic image based on the camera models and in the number N directions to obtain N sub-images of the panoramic image includes: Set the initial rotation angle θ of the first sub-image when segmenting the panoramic image; Based on the initial rotation angle θ, N cutting directions are determined according to the number of directions N; the interval angle β between the N cutting directions is (360° / N); the N cutting directions are: θ, θ+β, θ+2β, ... θ+(N-1)×β; The camera model is formed by combining the initial rotation angle, field of view parameters, N cutting directions, and preset resolution parameters. Based on the camera model, the panoramic image is projected onto the pixel coordinate plane to obtain N sub-images; Extraction unit, adapted to extract image feature data of the panoramic image sub-image; A storage unit is provided for storing the panoramic image data; the panoramic image data includes: the panoramic image, the sub-images corresponding to the panoramic image, and image feature data.

8. A system for constructing panoramic image data, based on the system as described in claim 7, characterized in that, The acquisition unit is also suitable for acquiring panoramic images p'; The acquisition unit is also adapted to acquire multiple sets of camera parameters to obtain multiple camera models based on when the panoramic image p' is segmented into multiple sub-images. The multiple sets of camera parameters include: field of view parameters α1, α2, α3, ... α z1 Resolution parameters ω1, ω2, ω3, ... ω z2 ; The computing unit is also adapted to calculate, based on the multiple sets of camera parameters, the number of multiple directions n1, n2, n3, ... n corresponding to segmenting the panoramic image p' into multiple sub-images. z3 ; The construction system further includes: a setting unit, adapted to set multiple initial rotation angles θ1, θ2, θ3, ... θ of the first sub-image when segmenting the panoramic image p'. z4 z1, z2, z3, and z4 are natural numbers greater than 1. The construction system further includes: a combination unit, adapted to combine the multiple sets of camera parameters and the multiple sets of direction quantities n1, n2, n3, ... n z Multiple initial rotation angles are used to combine the parameters of the camera model to obtain multiple sets of combined camera parameters and multiple sets of cutting directions; The construction system further includes: a building unit, adapted to build multiple camera models based on the combined multiple sets of camera parameters; The projection unit is further adapted to select a camera model from the multiple camera models and determine the corresponding cutting direction, projecting the panoramic image P' onto the pixel coordinate system plane to obtain a preset number of sub-images; let N0 sub-images obtained by projecting the panoramic image P' be q'1, q'2, q'3, ... q' N0 N0 is the preset number of sub-images obtained based on multiple camera models and corresponding cutting directions; N0 is a natural number greater than 1. The extraction unit is also adapted to extract sub-images q'1, q'2, q'3, ... q' from the panoramic image P'. N0 Image feature data F'1, F'2, F'3…F' N0 ; The storage unit is also adapted to store the panoramic image data; the panoramic image data includes: a panoramic image P', and N sub-images q'1, q'2, q'3, ... q' N0 And the image feature data F'1, F'2, F'3…F' corresponding to each sub-image. N0 .

9. A panoramic image data retrieval system, based on panoramic image data constructed by the construction system as described in claim 7 or 8, characterized in that, include: The receiving unit is adapted to receive the input image; The extraction unit is also adapted to extract image features from the input image; The acquisition unit is also adapted to acquire panoramic image data; The matching unit is adapted to match the image features of the input image with the image feature data of the panoramic image data to obtain the image feature data that is closest to the image features of the input image; The retrieval unit is adapted to retrieve the panoramic image most similar to the input image based on the matched image feature data.

10. A system for constructing panoramic image data, characterized in that, include: Processor and memory; The memory stores a computer program, and when the processor executes the computer program, it performs the steps of the method for constructing panoramic image data as described in any one of claims 1 to 4.

11. A panoramic image data retrieval system, characterized in that, include: Processor and memory; The memory stores a computer program, and when the processor executes the computer program, it performs the steps of the panoramic image data retrieval method as described in claim 5 or 6.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for constructing panoramic image data as described in any one of claims 1 to 4.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the panoramic image data retrieval method as described in claim 5 or 6.

Images