Target detection method for panoramic image

Abstract

The invention provides a target detection method for a panoramic image, and provides a method for reducing a storage space for hardware equipment, establishing lattice point mapping by taking lattice points as coordinates, then performing linear interpolation to establish coordinate mapping for projecting a panoramic equirectangular projection method into a 2D plane image through mapping, and further performing a target detection method. In order to improve the accuracy of a target detection algorithm, a panoramic image is projected into six planes during detection, and target detection is performed on each plane image.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic image processing, and in particular to a method for detecting objects in panoramic images. Background Art

[0002] Panoramic images, also known as 360° images or spherical images, can show a 360° view of the environment around the photography point and provide scene information with a large field of view. Users can capture images using devices such as fisheye lenses or panoramic cameras, and can fully present them on a spherical surface. In the image field of view, spherical images provide scenes within a range of 360°×180° at the viewpoint, providing more information than traditional 2D plane images, and their resolution is usually very high, but spherical images are not easy to store and transmit, and do not conform to human visual habits. Therefore, spherical images are usually converted into 2D plane images through some mapping methods after acquisition. Common panoramic images are usually plane images converted from spherical panoramic images by projection mapping. The projection of panoramic images refers to the conversion of spherical panoramic image content into 2D plane form, which is often used to encode spherical panoramic images for efficient processing, storage and transmission of panoramic images. Panoramic images are generally projected using equirectangular projections. Equirectangular panoramic images are obtained through equirectangular projections. Object detection is a commonly used type of detection in panoramic images, also known as object extraction. It is to accurately locate the categories and location information of various objects in the image and its sequence through identification boxes. At present, the mainstream object detection algorithms basically use deep learning methods. Traditional object detection methods generally use plane images for detection. Therefore, panorama object detection technology can be more suitable for panoramic VR scenes and has strong practical significance.

[0003] Due to the calculation method of the equirectangular projection method, the pixel distribution in the obtained equirectangular panoramic image is uneven, there is a lot of redundancy in the pixels at the upper and lower edge positions, and the pixels in the middle position are relatively sparse, and the image has distortion and distortion problems. Although there are many existing algorithms for image target detection tasks, and there are also excellent algorithms, most of them are aimed at conventional plane images with uniform image pixel distribution and target objects in the image with characteristics such as translation invariance and rotation invariance, but this is not applicable to equirectangular panoramic image data. Some target objects in panoramic images show asymmetry, that is, the pixels that make up the objects in the image are irregularly distributed in space, which causes the objects in the panoramic image to lose their invariant characteristics in the conventional image. Due to the difference in the basic structure of the two images, the algorithm model of the conventional image is directly used in the equirectangular panoramic image data. The model cannot extract the complete feature information of the object, resulting in many missed detections and false detections.

[0004] At the same time, panoramic images are usually high-resolution, with a resolution of 2048×1024 or above, which is beneficial for target detection tasks. However, in a large field of view environment, the targets in panoramic images are more likely to exist in the form of small objects, and the distribution of objects does not occupy the entire image, which is also difficult for detection tasks.

[0005] The invention patent application with application number CN114494824A discloses a method, device, equipment and storage medium for target detection of panoramic images. When performing target detection on a panoramic image, the current first panoramic image is rolled and translated horizontally so that the incomplete target objects in the edge area are spliced ​​to form a complete target object, thereby obtaining a translated second panoramic image, and then according to the rolling and translation relationship between the first panoramic image and the second panoramic image, a first target detection frame corresponding to the target detection frame in the edge area of ​​the first panoramic image is determined from the second panoramic image, and then the target object contained in the first panoramic image is determined according to the target detection frame in the non-edge area of ​​the first panoramic image and the first target detection frame in the second panoramic image. The disadvantage of this method is that it only splices the edge area and directly performs target detection on the panoramic image. Since some target objects in the panoramic image are asymmetric, that is, the pixel points constituting the object in the image are irregularly distributed in space, which causes the object in the panoramic image to be deformed, resulting in low detection accuracy. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a target detection method for panoramic images. For hardware devices, in order to reduce storage space, a grid mapping with grid points as coordinates is established, and then linear interpolation is performed to establish a coordinate mapping that projects the panoramic equirectangular projection method into a 2D plane image through mapping, and then a target detection method is further performed. In order to improve the accuracy of the target detection algorithm, the panoramic image is projected into 6 planes during detection, and target detection is performed on each plane image separately.

[0007] The present invention provides a method for detecting an object in a panoramic image, comprising acquiring a panoramic image, and further comprising the following steps:

[0008] Step 1: Projecting the panoramic image into a coordinate mapping of a 2D plane image by mapping using an equirectangular projection method;

[0009] Step 2: Sampling the continuous coordinates into n×n grid points to save the corresponding coordinates, making a hardware mapping table, and storing the hardware mapping table in the memory, where n is a constant;

[0010] Step 3: Read the hardware mapping table, perform linear interpolation, and derive all coordinates within the n×n grid;

[0011] Step 4: Project the panoramic image into 6 planes by mapping it into coordinates of a 2D plane image according to an equirectangular projection method;

[0012] Step 5: Perform target detection using a general algorithm on the six plane images;

[0013] Step 6: Determine where the target appears.

[0014] Preferably, the step 1 comprises the following sub-steps:

[0015] Step 11: Detect and extract features and key points of the panoramic image;

[0016] Step 12: Matching feature vectors between the panoramic image and the 2D plane image;

[0017] Step 13: Estimate the homography matrix using the RANSAC algorithm and the matched eigenvectors;

[0018] Step 14: Establish a mapping relationship between the equirectangular projection method and the 2D plane image according to the homography matrix.

[0019] In any of the above schemes, preferably, step 2 comprises the following sub-steps:

[0020] Step 21: Establish a mapping relationship table from panoramic images to 2D plane images;

[0021] Step 22: Sampling points on an n×n grid, saving the coordinates of the grid points, and generating a hardware mapping table.

[0022] In any of the above solutions, preferably, the mapping relationship table includes mapping relationships of all images.

[0023] In any of the above solutions, it is preferred that the horizontal coordinate of the grid point is x i , the n horizontal coordinates of linear interpolation are

[0024] Delta_x=(x2-x1) / n

[0025] x n =x1+Delta_x×j

[0026] Delta_x represents the scaling ratio of a single pixel coordinate in the horizontal direction, i = 0, 1, 2, ..., n, x1 represents the horizontal coordinate of the first grid point, x2 represents the horizontal coordinate of the second grid point, and x n Represents the horizontal coordinate of the nth grid point, j=0,1,2...n-1.

[0027] In any of the above schemes, it is preferred that the vertical coordinate of the grid point is yi , the n longitudinal coordinates of linear interpolation are

[0028] Delta_y=(x2-x1) / n

[0029] y n =y1+Delta_y×j

[0030] Among them, Delta_y represents the scaling ratio of a single pixel coordinate in the vertical direction, y1 represents the vertical coordinate of the first grid point, y2 represents the vertical coordinate of the second grid point, and y n Represents the vertical coordinate of the nth grid point.

[0031] In any of the above schemes, it is preferred that the grid point coordinates obtained are

[0032] (x1,y1), (x1+Delta_y, y1+Delta_y), (x1+Delta_x×2, y1+Delta_y×2),…, (x1+Delta_x×n-1, y1+Delta_y×n-1).

[0033] In any of the above schemes, preferably, step 6 includes performing normal target detection on 6 images respectively, and when a target appears at a boundary of a block in an image, remapping and dividing the adjacent image sizes are performed.

[0034] In any of the above solutions, preferably, step 6 further includes replacing the equal division rule with an unequal division rule according to the size of the frame occupied by the current target.

[0035] In any of the above schemes, it is preferred that the unequal division rule is to perform partial movement with n×n grid points, directly cover the target object in its entirety on the current plane, and then re-perform target detection on the newly divided 6 planes.

[0036] The present invention proposes a target detection method for panoramic images, which can greatly improve the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The flowchart is a preferred embodiment of the object detection method for panoramic images according to the present invention.

[0038] Figure 2 The figure is a schematic diagram of an embodiment of the relationship between a 2D plane image and a panoramic image according to the object detection method for panoramic images of the present invention.

[0039] Figure 3 FIG. 1 is a schematic diagram of a panoramic image according to an embodiment of a method for detecting an object in a panoramic image of the present invention.

[0040] Figure 4 The figure is a schematic diagram of sampling of a panoramic image according to an embodiment of the method for object detection in a panoramic image of the present invention.

[0041] Figure 5 The figure is a schematic diagram of an embodiment of projecting a panoramic image into six planes according to the method for object detection in a panoramic image of the present invention.

[0042] Figure 6 FIG. 1 is a schematic diagram of an embodiment of a method for detecting an object in a panoramic image according to the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0044] Embodiment 1

[0045] like Figure 1 As shown, step 100 is executed to obtain a panoramic image.

[0046] Executing step 110, using an equirectangular projection method to project the panoramic image into a coordinate mapping of a 2D plane image by mapping, includes the following sub-steps:

[0047] Execute step 111 to detect and extract features and key points of the panoramic image;

[0048] Executing step 112, matching feature vectors between the panoramic image and the 2D plane image;

[0049] Execute step 113, estimate the homography matrix using the RANSAC algorithm and the matched eigenvectors;

[0050] Execute step 114 to establish a mapping relationship between the equirectangular projection method and the 2D plane image according to the homography matrix.

[0051] Execute step 120, extract the continuous coordinates according to the n×n grid points to save the corresponding coordinates, make a hardware mapping table, and store the hardware mapping table in the memory, wherein n is a constant, including the following sub-steps:

[0052] Execute step 121 to establish a mapping relationship table from panoramic images to 2D plane images, wherein the mapping relationship table includes mapping relationships of all images.

[0053] Execute step 122 to sample points according to the n×n grid, save the coordinates of the grid points, and generate a hardware mapping table.

[0054] Execute step 130, read the hardware mapping table, perform linear interpolation, derive all coordinates within the n×n grid, and set the horizontal coordinate of the grid point as x i, the n horizontal coordinates of linear interpolation are

[0055] Delta_x=(x2-x1) / n

[0056] x n =x1+Delta_x×j

[0057] Delta_x represents the scaling ratio of a single pixel coordinate in the horizontal direction, i = 0, 1, 2, ..., n, x1 represents the horizontal coordinate of the first grid point, x2 represents the horizontal coordinate of the second grid point, and x n represents the horizontal coordinate of the nth grid point, j = 0, 1, 2...n-1;

[0058] Let the horizontal coordinate of the grid point be y i , the n longitudinal coordinates of linear interpolation are

[0059] Delta_y=(x2-x1) / n

[0060] y n =y1+Delta_y×j

[0061] Among them, Delta_y represents the scaling ratio of a single pixel coordinate in the vertical direction, y1 represents the vertical coordinate of the first grid point, y2 represents the vertical coordinate of the second grid point, and y n Represents the vertical coordinate of the nth grid point;

[0062] The grid point coordinates obtained are

[0063] (x1,y1), (x1+Delta_y, y1+Delta_y), (x1+Delta_x×2, y1+Delta_y×2),…, (x1+Delta_x×n-1, y1+Delta_y×n-1).

[0064] Step 140 is executed to project the panoramic image into six planes by mapping it into coordinates of a 2D plane image according to an equirectangular projection method.

[0065] Execute step 150 to perform target detection using a general algorithm on the six plane images.

[0066] Execute step 160 to determine the position where the target appears. Perform normal target detection on the six images respectively. When the target appears at the boundary of the block in the image, remap and divide the adjacent image sizes.

[0067] According to the size of the frame occupied by the current target, the equal division rule is replaced by the unequal division rule. The unequal division rule is to perform partial movement with n×n grid points, directly cover the target object in the current plane, and then re-perform target detection on the newly divided 6 planes.

[0068] Embodiment 2

[0069] The present invention proposes a method for object detection for panoramic images. Since panoramic images are imaged as spherical panoramas during acquisition, they are often projected into 2D plane images using an equirectangular projection method for ease of storage and transmission. This is also the most important form of expression for panoramic images. Figure 2 As shown, there is a pixel correspondence relationship between the 2D plane image and the panoramic image. The panoramic image is a panoramic image obtained by mapping the original image.

[0070] The traditional panoramic target detection method mainly performs deep learning neural network detection on the deformed VR image. Due to the distortion problem in the panoramic image, the detection algorithm of the conventional image is difficult to extract the complete information of the object in the image, and the target detection algorithm needs to be changed. Therefore, there are disadvantages such as algorithm incompatibility and low accuracy. Therefore, in the hardware implementation of the present invention, in order to reduce the storage space, a grid point mapping with grid points as coordinates is established, and then a linear interpolation is performed to establish a coordinate mapping that projects the panoramic equirectangular projection method into a 2D plane image through mapping, and then a target detection method is further performed. In order to improve the accuracy of the target detection algorithm, the panoramic image is projected into 6 planes during detection, and target detection is performed on each plane image respectively. In order to prevent the loss of the target at the junction of each plane, when the target appears in the image at the boundary of the block, the adjacent image size will be re-divided in this case, and the equal division rule will be replaced with the unequal division rule according to the size of the frame occupied by the current target. In actual operation, when the offset is performed according to a multiple of 16, the absolute position of the target in the stitched panoramic target image is calculated based on the offset of 16 and the relative coordinates of the target in each block. This method can greatly improve the accuracy of target detection. The main steps include:

[0071] 1. Establish an equirectangular projection method to project it into a coordinate mapping of a 2D plane image through mapping.

[0072] (1) Detect and extract image features and key points,

[0073] (2) Matching descriptors between two images,

[0074] (3) Use the RANSAC algorithm to estimate the homography matrix using our matched eigenvectors,

[0075] (4) Based on the homography matrix, the mapping relationship between the equirectangular projection method and the 2D plane image is established.

[0076] 2. Process the continuous coordinates in a square grid. In this embodiment, 8x8 is used as an example. The corresponding coordinates are saved by sampling the 8x8 grid points, and a hardware mapping table is made. The hardware mapping table is stored in the memory.

[0077] (1) Establish Figure 3 The mapping relationship between the panoramic image and the 2D plane image is shown, because this mapping table will include the mapping relationship of all images.

[0078] (2) Figure 4 As shown, the points are sampled in an 8x8 grid, and only the coordinates at the grid points are saved to make a hardware mapping table.

[0079] 3. Read the corresponding hardware mapping table, perform linear interpolation, and derive all coordinates within the 8x8 grid.

[0080] For example, if the two horizontal coordinates at the grid point are x1 and x8, the eight coordinates of linear interpolation are: Delta_x = (x2-x1) / 8

[0081] X n =x1+Delta_x*i(i=0,1,2,...,7)

[0082] So the coordinates are:

[0083] x1,x1+Delta_x,x1+Delta_x*2,x1+Delta_x*3,x1+Delta_x*4,x1+Delta_x*5,x1+Delta_x*6,x1+Delta_x*7.

[0084] A similar approach is used in the Y direction

[0085] For example, the two vertical coordinates at the grid point are y1 and y8, and the 8 coordinates of linear interpolation are: Delta_y = (y2-y1) / 8

[0086] Y n =y1+Delta_y*i(i=0,1,2,...,7)

[0087] Therefore, the coordinates are: y1, y1+Delta_y, y1+Delta_y*2, y1+Delta_y*3, y1+Delta_y*4, y1+Delta_y*5, y1+Delta_y*6, y1+Delta_y*7.

[0088] 4. According to the equirectangular projection method, the panoramic image is projected into 6 planes by mapping it into a 2D plane image coordinate mapping. Figure 5 shown.

[0089] 5. If Figure 6 As described above, target detection using a general algorithm is performed on 6 plane images.

[0090] 6. When normal target detection is performed on 6 images, the target may appear at the boundary of two images. When the target appears at the boundary of the block in the image, the adjacent image size is remapped and divided. According to the size of the frame occupied by the current target, the bisection rule is replaced with an unequal division rule. Specifically, the 8x8 grid is used for partial movement, and the current plane directly covers the target object. Then the target detection method is re-performed on the newly divided 6 planes.

[0091] The technical effects that can be achieved by the present invention are:

[0092] 1. Establish an equirectangular projection method to project it into a coordinate mapping of a 2D plane image through mapping, so that general target detection algorithms can be used universally without the need to retrain algorithms and models, thereby improving algorithm utilization and reducing R&D costs.

[0093] 2. When implementing hardware design, in order to reduce storage space, a grid-based coordinate mapping is established, and then linear interpolation is performed to establish a coordinate mapping that projects the panoramic equirectangular projection method into a 2D plane image through mapping, thereby reducing bandwidth and hardware costs.

[0094] 3. When performing target detection on each plane image separately, in order to prevent the target from being lost at the junction of each plane, the invention re-divides the adjacent image size when the target appears at the boundary of the block in the lower image, and replaces the equal division rule with the unequal division rule according to the size of the frame occupied by the current target. This method can greatly improve the accuracy of target detection.

[0095] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A method for object detection in a panoramic image, comprising acquiring a panoramic image, characterized in that: The following steps are also included: Step 1: Projecting the panoramic image into a coordinate mapping of a 2D plane image by mapping using an equirectangular projection method; Step 2: Sampling the continuous coordinates into n×n grid points to save the corresponding coordinates, making a hardware mapping table, and storing the hardware mapping table in memory, where n is a constant; Step 3: Read the hardware mapping table, perform linear interpolation, and derive all coordinates within the n×n grid; Step 4: Project the panoramic image into 6 planes by mapping it into coordinates of a 2D plane image according to an equirectangular projection method; Step 5: Perform target detection using a general algorithm on the six plane images; Step 6: Determine where the target appears.

2. The object detection method for panoramic images according to claim 1, characterized in that: The step 1 includes the following sub-steps: Step 11: Detect and extract features and key points of the panoramic image; Step 12: Matching feature vectors between the panoramic image and the 2D plane image; Step 13: Estimate the homography matrix using the RANSAC algorithm and the matched eigenvectors; Step 14: Establish a mapping relationship between the equirectangular projection method and the 2D plane image according to the homography matrix.

3. The object detection method for panoramic images according to claim 2, characterized in that: The step 2 includes the following sub-steps: Step 21: Establish a mapping relationship table from panoramic images to 2D plane images; Step 22: Sampling points on an n×n grid, saving the coordinates of the grid points, and generating a hardware mapping table.

4. The object detection method for panoramic images according to claim 3, characterized in that: The mapping relationship table includes mapping relationships of all images.

5. The object detection method for panoramic images according to claim 4, characterized in that: Let the horizontal coordinate of the grid point be x i , the n horizontal coordinates of linear interpolation are Delta_x=(x2-x1) / n x n =x1+Delta_x×j Delta_x represents the scaling ratio of a single pixel coordinate in the horizontal direction, i = 0, 1, 2, ..., n, x1 represents the horizontal coordinate of the first grid point, x2 represents the horizontal coordinate of the second grid point, and x n Represents the horizontal coordinate of the nth grid point, j=0,1,2...n-1.

6. The object detection method for panoramic images according to claim 5, characterized in that: Let the vertical coordinate of the grid point be y i , the n longitudinal coordinates of linear interpolation are Delta_y=(x2-x1) / n and n =y1+Delta_y×j Among them, Delta_y represents the scaling ratio of a single pixel coordinate in the vertical direction, y1 represents the vertical coordinate of the first grid point, y2 represents the vertical coordinate of the second grid point, and y n Represents the vertical coordinate of the nth grid point.

7. The object detection method for panoramic images according to claim 6, characterized in that: The obtained grid point coordinates are (x1, y1), (x1+Delta_y, y1+Delta_y), (x1+Delta_x×2, y1+Delta_y×2),…, (x1+Delta_x×n-1, y1+Delta_y×n-1).

8. The object detection method for panoramic images according to claim 7, characterized in that: The step 6 includes performing normal target detection on the six images respectively. When a target appears at the boundary of a block in an image, the adjacent image sizes are remapped and divided.

9. The object detection method for panoramic images according to claim 8, characterized in that: The step 6 also includes replacing the equal division rule with an unequal division rule according to the size of the frame occupied by the current target.

10. The object detection method for panoramic images according to claim 9, characterized in that: The rule of the unequal division is to perform partial movement with n×n grid points, directly cover the target object in its entirety on the current plane, and then perform target detection again on the newly divided 6 planes.

Images