Position sensing system based on panoramic image deep learning

Abstract

The application discloses a location perception system based on panoramic image deep learning, which utilizes image synthesis technology, digital image processing technology and deep learning technology for location perception. The location perception system comprises a panoramic camera, an image processing subsystem, a panoramic image database, a region perception subsystem and a positioning subsystem. The region perception subsystem divides a region into multiple regions, takes pictures in the panoramic image database as samples, performs deep learning training, and obtains a region perception classification model. The positioning subsystem compares feature points of a panoramic image output by the image processing subsystem with feature points of images in the region based on a classification result of the region subsystem, so that location perception is realized. The panoramic camera with wireless communication function only needs to be carried by a positioning target, so that accurate positioning can be realized. The system has simple structure and is easy to implement, and can be widely applied to the field of accurate positioning of local space, such as schools, hospitals, factories, coal mines, subways and the like, and has wide application and popularization space.

Description

Technical Field

[0001] This invention relates to a location awareness system based on deep learning of panoramic images, and the method involves panoramic images, deep learning, image processing, and spatial positioning technology. Background Technology

[0002] Currently, open-space mobile target positioning primarily relies on satellite positioning, including GPS, GLONASS, Galileo, and BeiDou systems. In addition, positioning technologies based on cellular mobile communication systems, Wi-Fi networks, and wireless sensor networks are also used. However, cellular mobile communication system positioning is generally only used for business management by telecommunications operators and does not directly provide personal positioning services to users; positioning based on Wi-Fi networks and wireless sensor networks can only achieve positioning within a localized area. Therefore, satellite positioning remains the primary positioning technology relied upon in people's lives. However, satellite positioning requires unobstructed skies to operate, greatly limiting its application; moreover, when satellites malfunction or satellite positioning services are shut down, positioning becomes impossible, causing significant disruption.

[0003] In enclosed environments such as buildings, tunnels, subway stations, and underground coal mines where satellite signals cannot be received, positioning is often based on RFID card identification, wireless signals, and images. RFID card identification and positioning technology utilizes radio frequency signals for contactless two-way communication, enabling target identification and location monitoring without physical contact between the RFID card and the reader. However, RFID card identification and positioning technology is a regional positioning technology, only able to identify whether a moving target has passed through a certain area, but unable to accurately locate the position of moving objects within that area. Wireless signal positioning technologies mainly include positioning methods based on signal attenuation (RSSI) and time of arrival (TDOA). Because wireless signals are easily affected by factors such as the size, shape, and wall roughness of enclosed environments during transmission, the wireless signal attenuation model is extremely complex. Positioning systems based on radio signal transmission time have higher positioning accuracy than RSSI positioning systems, but radio signal transmission time is affected by multipath effects, non-line-of-sight propagation delays, clock synchronization, and clock timing errors, making it difficult to guarantee positioning accuracy in environments with many obstacles. Furthermore, current positioning methods such as RFID, RSSI, Time of Flight (TOA), and Time Difference of Flight (TDOA) are all based on ranging principles, requiring the deployment of a large number of receiving devices in the space beforehand, resulting in high deployment costs and difficult maintenance. Image-based localization mainly includes label-based localization and semantic segmentation-based localization. Label-based localization requires pre-deploying labels with location information in the space to obtain more environmental information for easier localization; however, this method requires deploying a large number of labels in advance, resulting in high deployment costs. Semantic segmentation-based localization requires pre-labeling objects with location information in the environment and recording their coordinates. When three or more labeled objects are identified, a coordinate matrix is ​​established based on the object coordinates and camera intrinsic parameters to calculate the image's position coordinates; however, this method requires extensive image labeling, and the coordinate calculation is cumbersome.

[0004] Therefore, it is essential to have a positioning system that is simple, effective, low-cost, and highly accurate, suitable for confined spaces such as coal mines, and capable of achieving precise three-dimensional positioning, including the orientation of the target under static conditions. Summary of the Invention

[0005] Compared to monocular cameras, binocular cameras can acquire more environmental information. Panoramic cameras, through image stitching, can obtain 360° environmental images with more image features. This invention combines panoramic cameras, deep learning, and digital image processing technologies to propose a position awareness system based on deep learning of panoramic images, achieving region determination and precise local spatial positioning. The system includes a panoramic camera, an image processing subsystem, a panoramic image database, a region awareness subsystem, and a positioning subsystem. The panoramic camera has two fisheye lenses (front and rear). The image processing subsystem stitches images from different perspectives captured by the camera at the same location to generate a panoramic image. The panoramic image database consists of panoramic images from different viewpoints and their coordinates; the panoramic images include planar panoramic images and spherical panoramic images. The region awareness subsystem divides the environment into multiple regions and uses the panoramic images as samples for deep learning training to obtain a region awareness classification model. The positioning subsystem, based on the classification results of the region awareness subsystem, compares the feature points of the panoramic image output by the image processing subsystem with the feature points of the images within that region to achieve position awareness.

[0006] The image processing steps of the system are as follows:

[0007] (1) Feature point matching: Label the regions of interest (ROI) of the two images taken by the fisheye lens. Select the point to be processed in the ROI of one image and calculate the Euclidean distance from the point to each point in the ROI of the other image. If the ratio of the minimum Euclidean distance to the second minimum Euclidean distance is less than the set threshold, the point is coarsely matched with the point with the smallest Euclidean distance. For points that fail to be coarsely matched, the RANSAC algorithm is used to match again.

[0008] (2) Pixel fusion: The pixel values ​​of the ROI regions of the two images to be stitched are weighted and averaged to determine the pixel value of the stitching point, so as to eliminate the brightness difference caused by the different ambient light of the two fisheye lenses and improve the correlation between the stitching regions.

[0009] The panoramic image database acquisition method of the system includes:

[0010] (1) Fixed camera position acquisition: Fix the panoramic camera at a certain position, record the coordinates of that position in the environment, capture panoramic video at that position, and obtain a panoramic image by extracting video image frames;

[0011] (2) Moving camera position acquisition: Place the panoramic camera on a moving object, and the moving object moves in a single direction to capture panoramic video of the environment. The position information of the panoramic camera is obtained by calculation, and the panoramic image is obtained by extracting video image frames.

[0012] The training steps for the region-aware subsystem of the system are as follows:

[0013] (1) Divide the environment to be located into multiple spatial regions, record the coordinate range of each region, and mark the region where the panoramic image in the panoramic image database is located.

[0014] (2) Take each region in step (1) as a class, and take the panoramic image in each region as a dataset for deep learning, and feed it into the classification network for training.

[0015] (3) Repeat step (2) to obtain the region perception model.

[0016] The environmental perception steps of the region perception subsystem of the system are as follows:

[0017] (1) The panoramic camera inputs the panoramic image it has acquired into the region perception model trained in claim 4. The region perception model outputs a tensor as a feature description of the image.

[0018] (2) Compare the feature description of the image with the feature description of each region image, and take the region with the highest matching degree as the region where the image is located;

[0019] (3) Transmit the area perception results to the positioning subsystem.

[0020] The positioning steps of the regional positioning subsystem of the system are as follows:

[0021] (1) The panoramic camera inputs the panoramic image it has acquired into the region perception model trained in claim 4. The region perception model outputs a tensor as a feature description of the image.

[0022] (2) Compare the feature description of the image with the feature description of each region image, and take the region with the highest matching degree as the region where the image is located;

[0023] (3) Transmit the area perception results to the positioning subsystem.

[0024] The positioning steps of the positioning subsystem are as follows:

[0025] (1) Based on the region matching results, select the panoramic image in the region as the object to be matched;

[0026] (2) Extract the feature points of the panoramic image to be located, and calculate the number of matching feature points between the panoramic image to be located and each panoramic image in the region according to the planar mapping relationship;

[0027] (3) Select the panoramic image with the most feature point matching in the region as the localization result.

[0028] 1. The location sensing system further includes: a deployment platform for the location sensing system including a microcontroller, an FPGA, and a DSP.

[0029] 2. The location-aware system further includes: an image processing subsystem that calculates feature points X(x1, x2, ... x...). n ) and feature points Y(y1,y2,…y n Euclidean distance between ) The formula for calculating the pixel value of the blended region is: Pixel = K·Pixel F +(1-K)·Pixel B Pixel F and Pixel B K = S, where S represents the pixel values ​​of the areas to be stitched together from the front and rear fisheye lens images, respectively. F / (S F +S B ), S F S B These represent the Euclidean distances from the front and rear fisheye lens images to the points in the region to be stitched, respectively.

[0030] 3. The location sensing system further includes: panoramic images in the panoramic image database, including planar panoramic images and spherical panoramic images; planar panoramic images can be stitched together to obtain spherical panoramic images.

[0031] 4. The location sensing system further includes: a method for acquiring the panoramic database (1) the panoramic images are set on a horizontal plane with a height of h, with a minimum interval of gap. min In method (2), the moving object moves at a constant speed v. Based on the video duration t where the image is located, the moving distance d = v·t is calculated to obtain the coordinates of the panoramic image.

[0032] 5. The location sensing system further includes: when the area sensing subsystem senses multiple areas with the highest matching degree in step (2), the result of step (2) is sent to the positioning subsystem for feature point matching, and the area with the highest feature point matching degree is taken as the positioning result.

[0033] 6. The location-aware system further includes: feature point matching methods including SFIT, ORB, BING, and Geodesc algorithms.

[0034] The beneficial effects achieved by this invention are as follows: This invention combines image synthesis technology, digital image processing technology, and deep learning technology for location perception. The target only needs to carry a panoramic camera with wireless communication capabilities to achieve precise positioning. The system has a simple structure, is easy to implement, and can be widely applied in the field of precise local spatial positioning, such as in schools, hospitals, factories, coal mines, and subways, with broad application and promotion potential. Attached Figure Description

[0035] Figure 1 Schematic diagram of a location awareness system based on deep learning from panoramic images.

[0036] Figure 2 A schematic diagram of the deep learning process for an environmental perception system.

[0037] Figure 3 Example diagram of environmental zoning.

[0038] Figure 4 Schematic diagram of the positioning subsystem workflow. Detailed Implementation

[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0040] Figure 1 This is a flowchart of the location awareness system based on deep learning of panoramic images. Figure 1 As shown, the location awareness system in this embodiment includes an offline component (panoramic camera (101), panoramic image database (102), area awareness subsystem (103), monitoring terminal (104)) and an online component (panoramic camera (105), image processing subsystem (106)(107), positioning subsystem):

[0041] 1. Panoramic camera (101)(105) for capturing panoramic video, used to generate a panoramic image database and for positioning.

[0042] 2. Panoramic image database (102): Extract video image frames captured by panoramic camera (101), record the coordinate information of each image, and generate a panoramic image database.

[0043] 3. Region Awareness Subsystem (103): This subsystem trains a region classification model using images from a panoramic image database and then transmits the awareness model to the localization subsystem. The environment is divided into multiple regions, the coordinate range of each region is determined, and region annotations are performed on images within the image dataset. A region awareness model is trained using a classification network and then transmitted to the localization subsystem.

[0044] 4. The monitoring terminal (104) is mainly responsible for the equipment management of the area perception subsystem, panoramic camera, and positioning subsystem. Management personnel can access these devices through the monitoring terminal and manage their parameters and stored data. This includes tasks such as extracting video image frames, annotating, deleting, and modifying images in the panoramic image database. Depending on system application needs, moving target monitoring software can be installed on the monitoring terminal to display images captured by the panoramic camera and monitor the position of moving targets.

[0045] 5. Image stitching (106) and image preprocessing: Process the images acquired by the panoramic camera;

[0046] 6. The localization subsystem (108) is used to locate the specific location of the image; it receives the environment perception model, inputs the preprocessed image into the perception model, and determines the region to which it belongs; then it performs feature point matching between the image and the images in the region to which it belongs, and selects the image with the highest matching degree as the localization result.

[0047] The deep learning process of the environmental perception subsystem is shown in Figure 2, including:

[0048] 1. (201) Measure the three-dimensional geographic information of the environment, establish a coordinate system, and divide the set space into multiple spatial regions;

[0049] 2. (202) Label the panoramic images in the panoramic image database. The labeling content includes the region where the image is located and its coordinates;

[0050] 3. (203) Determine whether the number of labeled samples meets the standard. If it does, execute (204) and use the remaining samples as the test set. Otherwise, return to execute (203).

[0051] 4. (204) Using the region determined by (201) as a unit, the images in the region are used as samples for deep learning training. In this example, the deep learning network uses the GoogLeNet convolutional neural network.

[0052] 5. (205) Determine whether the training accuracy meets the standard. If it does, proceed to (206); if it does not, return to (206) and further optimize the network parameters.

[0053] 6. (206) Transmit the image classification and recognition model of all regions and the location data corresponding to all classes to the positioning subsystem.

[0054] Figure 3 This is an example diagram of region division. Regions can be divided unevenly; in this example, the nine regions are rectangular regions of varying lengths and widths, separated by gaps. min Image data is acquired for a square grid with sides of length h. The panoramic camera has a height of h and a positioning accuracy of gap. min .

[0055] The system location awareness process is as follows: Figure 4 As shown, it includes:

[0056] 1. (401) A panoramic camera captures environmental images.

[0057] 2. (402) The region-aware model receives the preprocessed image, and the processing methods include: filtering, edge enhancement, brightness adjustment, contrast adjustment, hue adjustment and saturation adjustment.

[0058] 3. (403) Perform convolution processing on the image input region perception model, and compare the output feature description with the feature description of each region.

[0059] 4. (404) Select the region with the highest matching degree as the matching result. If the matching result is not unique, it is re-entered into the classification model for matching again.

[0060] 5. (406) The localization subsystem performs feature point matching between the image to be matched and the panoramic image of the area where it is located, based on the classification results transmitted by (405), and selects the one with the highest matching degree as the localization result.

[0061] 6. (408) The positioning server determines the location of the panoramic camera based on the matching result obtained in (407).

[0062] 7. (409) The positioning subsystem transmits the position of the panoramic camera obtained in (408) to the monitoring terminal that needs location monitoring service or the intelligent device that provides location navigation service, and returns (401).

Claims

1. A location-aware system based on deep learning of panoramic images, characterized in that: The system includes a panoramic camera, an image processing subsystem, a panoramic image database, a region perception subsystem, and a positioning subsystem; the panoramic camera is a dual fisheye camera with front and rear fisheye lenses; the image processing subsystem stitches together the images captured by the front and rear fisheye lenses of the camera to generate a panoramic image; The panoramic image database consists of panoramic images and their coordinates at different locations. The panoramic images include planar panoramic images and spherical panoramic images. The region perception subsystem divides the environment into multiple regions, uses panoramic images as samples, performs deep learning training, and obtains a region perception classification model. The localization subsystem, based on the classification results of the region perception subsystem, compares the feature points of the panoramic image to be located with the feature points of the image in the region to achieve location perception. The steps of image processing are as follows: (1) Feature point matching: Label the regions of interest (ROI) of the two images taken by the fisheye lens. Select the point to be processed in the ROI of one image and calculate the Euclidean distance from the point to each point in the ROI of the other image. If the ratio of the minimum Euclidean distance to the second minimum Euclidean distance is less than the set threshold, the point is coarsely matched with the point with the smallest Euclidean distance. For points that fail to be coarsely matched, the RANSAC algorithm is used to match again. (2) Pixel fusion: The pixel values ​​of the ROI regions of the two images to be stitched are weighted and averaged to determine the pixel value of the stitching point, so as to eliminate the brightness difference caused by the different ambient light of the two fisheye lenses and improve the correlation between the stitching regions. The training steps for the region perception subsystem are as follows: (1) Divide the environment to be located into multiple spatial regions, record the coordinate range of each region, and mark the region where the panoramic image in the panoramic image database is located; (2) Take each region in step (1) as a class, and use the panoramic images in each region as the dataset for deep learning, and feed them into the classification network for training; (3) Repeat step (2) to obtain the region perception model; The environmental perception steps of the area perception model are as follows: (1) The panoramic camera inputs the panoramic image it has acquired into the trained region perception model. The region perception model outputs a tensor as a feature description of the image. (2) Compare the feature description of the image with the feature description of each region image, and take the region with the highest matching degree as the region where the image is located; (3) Transmit the area perception results to the positioning subsystem; The positioning steps of the positioning subsystem are as follows: (1) Based on the region matching result obtained in the environmental perception step (2) of the region perception model, select the panoramic image in the region as the object to be matched; (2) Extract the feature points of the panoramic image to be located, and calculate the number of matching feature points between the panoramic image to be located and each panoramic image in the region according to the planar mapping relationship; (3) Select the panoramic image with the most feature point matches in the region as the localization result; Feature points With feature points Euclidean distance between Pixel values ​​of the fused region The calculation formula is: ,in and These represent the pixel values ​​of the areas to be stitched together from the front and rear fisheye lens images, respectively. , , These represent the Euclidean distances from the front and rear fisheye lens images to the points in the region to be stitched, respectively.

2. The location sensing system as described in claim 1, characterized in that: The panoramic image acquisition methods in the panoramic image database include: fixed-position acquisition, which involves fixing a panoramic camera at a certain location, recording the coordinates of that location in the environment, capturing panoramic video at that location, and obtaining a panoramic image by extracting video image frames.

3. The location sensing system as described in claim 1, characterized in that: The panoramic image acquisition methods in the panoramic image database include: moving camera acquisition, which involves placing a panoramic camera on a moving object, the moving object moving in a single direction to capture panoramic video of the environment, calculating the position information of the panoramic camera, and extracting video image frames to obtain a panoramic image.

4. The location sensing system as described in claim 1, characterized in that: The deployment platform for a location-aware system includes: microcontroller, FPGA, and DSP.

5. The location sensing system as described in claim 2, characterized in that: The panoramic images in the panoramic image database include planar panoramic images and spherical panoramic images; planar panoramic images can be stitched together to obtain spherical panoramic images.

6. The location sensing system as described in claim 2, characterized in that: The captured panoramic images are set at an altitude of [height missing]. On the horizontal plane, with the minimum interval as Collect the data.

7. The location sensing system as described in claim 3, characterized in that: Moving objects at speed Uniform motion, based on the video duration in which the image is located. Calculate the distance it moves, d = v·t, to obtain the coordinates of the panoramic image.

8. The location sensing system as described in claim 1, characterized in that: If there are multiple regions with the highest matching degree in the environmental perception step (2) of the region perception model, the result of step (2) is sent to the positioning subsystem for feature point matching, and the region with the highest feature point matching degree is taken as the positioning result.

9. The location sensing system as described in claim 1, characterized in that: The feature point matching in the positioning step (2) of the positioning subsystem can use the SFIT, ORB, BING, and Geodesc algorithms.

Images