An indoor skin base station detection and positioning method and device based on end-cloud cooperation

Abstract

The application discloses a kind of indoor skin base station detection and positioning method and device based on end cloud cooperation, comprising: first using binocular camera to collect the picture to be detected;Then the picture is transmitted into end side platform for depth calculation;After that, the video frame is sent into buffer, and the first video frame of buffer is sent into cloud side platform for target detection and feature extraction;Boundary box, feature point and label are transmitted back to end side platform for tracking, and detection and positioning result is rendered on display device.The application will adaptively judge whether the current video frame is sent to cloud for detection according to cumulative tracking error, gray difference value between current frame and last frame, the number of video frames that have been tracked and the number of video frames that have been sent for detection but have not returned result in tracking process, so as to maximize the use of detection resources.The application adopts end cloud cooperation mode, which meets the requirement of smooth display picture, reduces the energy consumption of embedded device and solves the problem of engineering landing.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of target detection and positioning, and particularly relates to an indoor skin base station detection and positioning method and device based on end-cloud cooperation. BACKGROUND

[0002] With the rapid development of communication technology, 5G gradually enters people's life, and applications based on 5G technology also emerge in an endless stream. One of the important applications of 5G is to use indoor skin base stations to position user terminals, but this puts forward a small error requirement for the coordinate measurement of skin base stations. However, since the skin base station is laid out with the purpose of covering the current scene as comprehensively as possible, the distribution of the skin base station in the room does not have regularity, and the coordinate measurement of the base station is difficult, and the traditional method will waste a lot of labor and time cost.

[0003] The existing skin base station positioning method mainly uses manual laser ranging equipment positioning, and the error is large, because when the laser equipment is far away from the skin base station, the angle error of 1 degree laser line will cause 3-4 meter error of the skin base station coordinate. Therefore, it is necessary to design a skin base station detection and positioning method, which can automatically identify and position multiple skin base stations in the picture, improve the work efficiency of the measurement personnel, and reduce the coordinate measurement error. However, the existing target recognition and positioning device has the following problems:

[0004] 1. The traditional target recognition and positioning is usually deployed in a desktop computer, which is not suitable for the scene of skin base station positioning. For example, dozens of skin base stations will be deployed on one floor in a shopping mall, which requires the whole device to move constantly to position all the skin base stations, and the desktop equipment obviously cannot meet the requirement.

[0005] 2. The use of a movable embedded device can meet the skin base station positioning scene, but the target detection algorithm puts forward a high requirement for the hardware device, and the embedded device has limited computing power, which consumes a long time to execute one detection, resulting in low real-time performance of the whole device and poor user experience.

[0006] 3. The traditional end-cloud cooperation mode is to unload the target detection part to the cloud through the cooperation of the end side and the cloud side, and then send the result back to the end side. However, when the detection frame rate is high, the network bandwidth occupation is large, and the result sent back to the end side has a time delay, which will cause inaccurate positioning result. SUMMARY

[0007] The purpose of the present application is to provide an indoor skin base station detection and positioning method and device based on end-cloud cooperation to solve the problems existing in the prior art.

[0008] To achieve the above purpose, the present application provides an indoor skin base station detection and positioning method based on end-cloud cooperation, which comprises:

[0009] collecting a skin base station image, performing depth estimation on the skin base station image to obtain a depth map;

[0010] obtaining a bounding box, a target label and a target feature point of the skin base station image on the cloud side, sending the target tracking to the end side, calculating a tracking cumulative error, a frame difference value and a number of tracked video frames in the tracking process, setting a submission condition, and respectively judging whether the submission condition is met, if yes, sending the next frame image to the cloud side for information acquisition, performing KLT tracking, and if not, calculating a final coordinate of the skin base station according to the depth map and the bounding box, and displaying a skin base station positioning result.

[0011] Optionally, the process of collecting the skin base station image comprises: fixing a binocular camera on a four-wheel cart, and moving the four-wheel cart at a constant speed to collect the skin base station image.

[0012] Optionally, the process of depth estimation comprises: performing gradient filtering on the skin base station image by using a Sobel operator, performing cost operation, replacing the cost value of each pixel point with the total sum of the cost values in the field, and using dynamic programming to obtain the minimum optimal disparity of each pixel point after the replacement processing and conversion to obtain the depth map.

[0013] Optionally, the process of target tracking comprises: constructing a target detection model based on a YOLOv5s network, adding a 160*160 detection feature map in the target detection model, performing up-sampling on the feature map after the 17th layer in the head of the model, and adding a small target detection layer in the detection layer; and using the added model to perform target tracking.

[0014] Optionally, the standard deviation of the moving distance of all feature points of the skin base station between the current frame and the previous frame is taken as the tracking cumulative error; the difference value of the gray values of the corresponding pixel points between the current frame and the previous frame is taken as the frame difference value; and the number of tracked videos is counted, and the number is increased by one for each tracked frame.

[0015] Optionally, the process of calculating the final coordinate comprises: calculating the relative coordinate of the skin base station according to the bounding box and the depth map, obtaining the absolute coordinate of the skin base station through coordinate system conversion, setting an absolute coordinate error preset value, if the error does not exceed the preset value, considering the skin base station as the same, and taking the weighted average result of the absolute coordinates obtained from the continuous multiple frames of images as the final coordinate of the skin base station.

[0016] The application also provides an indoor skin base station detection and positioning device based on end-cloud cooperation, comprising:

[0017] an end side platform composed of an embedded computing device, and a cloud side platform composed of a cloud server.

[0018] Optionally, the end side platform comprises:

[0019] The image acquisition module is composed of a binocular camera and a four-wheel intelligent car, and is used for acquiring images of indoor base stations;

[0020] The depth estimation module is used for depth estimation of the images to obtain depth information;

[0021] The video frame buffer module is used for storing the acquired video frame sequence;

[0022] The adaptive submission module is used for adaptively submitting video frames meeting conditions to the cloud side platform for detection in the tracking process according to tracking errors, a difference value between a current frame and a previous frame, and a number of tracked video frames, so as to reduce network bandwidth occupation caused by frame-by-frame detection while ensuring accuracy of the base station.

[0023] The frame uploading module is used for uploading the first frame when the device starts and a next frame of the video frame determined by the first determination module, the second determination module and the third determination module in the adaptive submission module into the deep learning model for detection;

[0024] The tracking module is used for tracking the base station according to feature points of the base station;

[0025] The coordinate output module is used for converting relative coordinates of the base station in the picture into absolute coordinates, and then performing weighted average on a plurality of absolute coordinates as a final result.

[0026] The result rendering module is used for highlighting the base station in the picture and its final coordinates.

[0027] Optionally, the adaptive submission module comprises:

[0028] The first calculation module is used for calculating a cumulative tracking error of each video frame;

[0029] The first determination module is used for determining a video frame with a cumulative tracking error greater than a threshold value;

[0030] The second calculation module is used for calculating a difference value between a current video frame and a previous video frame;

[0031] The second determination module is used for determining a video frame with a difference value greater than a threshold value;

[0032] The third calculation module is used for calculating a number of tracked video frames;

[0033] The third determination module is used for determining a video frame with a number of tracked video frames equal to a threshold value;

[0034] The submission limiting module is used for calculating a number of video frames submitted for detection but without returned results, and limiting submission of the video frames if the number is greater than a threshold value.

[0035] Optionally, the cloud-side platform includes:

[0036] The AI ​​inference module is used to perform target detection and feature extraction on the images transmitted from the frame upload module, and to transmit the labels, feature points, and bounding boxes of the pico base station to the tracking module.

[0037] The technical effects of this invention are as follows:

[0038] This invention employs an edge-cloud collaborative model, which reduces the power consumption of embedded devices while meeting the requirements for smooth display, thus solving the problem of engineering implementation. Using this method can significantly improve the efficiency and accuracy of indoor pico base station coordinate measurement. Attached Figure Description

[0039] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a flowchart of an indoor pico base station detection and positioning method based on end-to-cloud collaboration in an embodiment of the present invention;

[0041] Figure 2 This is a flowchart illustrating the test submission condition determination in an embodiment of the present invention;

[0042] Figure 3 This is a flowchart of an indoor pico base station detection and positioning device based on end-to-cloud collaboration in an embodiment of the present invention;

[0043] Figure 4 This is a structural diagram of the adaptive inspection module in an embodiment of the present invention. Detailed Implementation

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0046] Example 1

[0047] like Figures 1-4 As shown, this embodiment provides an indoor pico base station detection and positioning method and apparatus based on edge-cloud collaboration, including:

[0048] Step one: the four-wheel intelligent car moves steadily in the scene at a uniform speed, and collects the image of the skin base station.

[0049] Step two: the end side calculates the depth of the collected image, and then sends the image to the cloud side for detection using YOLOv5s.

[0050] Step three: the cloud side sends the bounding box, target label, and target feature point back to the end side, and the end side performs KLT tracking on the skin base station.

[0051] Step four: during the tracking process, the tracking cumulative error, frame difference value, and the number of tracked video frames are continuously calculated. If the inspection condition is met, the next frame is sent to the cloud side for detection, and step three is repeated. If the inspection condition is not met, the final coordinates of the skin base station are calculated.

[0052] Further, step one includes the following sub-steps:

[0053] Sub-step 1.1: fix the binocular camera on the four-wheel car to ensure that the binocular camera picture is stable during the stable movement of the four-wheel car.

[0054] Sub-step 1.2: start the four-wheel car to take pictures during movement, and transfer the pictures to the embedded device for subsequent steps.

[0055] Further, step two includes the following sub-steps:

[0056] Sub-step 2.1: depth estimation is performed on the image. First, the Sobel operator is used to perform gradient filtering on the image, which can retain more edge and detail information. Then, the cost is calculated, and then the cost value of each pixel point is replaced with the sum of the cost values of the neighborhood. Finally, the minimum optimal disparity of each pixel point is obtained through dynamic programming, and the depth map is converted.

[0057] Sub-step 2.2: send the image to the cloud side through webRTC.

[0058] Further, the target detection in step three is expanded as follows:

[0059] Due to the small size of the skin base station, a small target detection layer is added to the original YOLOv5s network. The specific method is as follows: 1) The original model has three scales of detection feature maps, 20*20, 40*40, and 80*80. This application increases the detection feature map of 160*160 for detecting small targets of 4*4. 2) A step of upsampling is added to the feature map after the 17th layer in the model head, so that the feature map continues to expand, and then it is concatenated with the feature map of the second layer in the backbone network to obtain a larger feature map for small target detection. 3) A small target detection layer is added to the detection layer, a total of four layers [21, 24, 27, 30].

[0060] Further, referring to Figure 2 , step four includes the following sub-steps:

[0061] Sub-step 4.1, calculate the tracking cumulative error and determine the video frame when it is greater than threshold Th1. The standard deviation D of the moving distance of all m feature points of the skin base station between the current video frame and the previous video frame is calculated in this application std As the tracking cumulative error, the formula is as follows. Where d k is the moving distance of the current feature point, is the average of the moving distance of all feature points, and m is the number of feature points.

[0062]

[0063] Sub-step 4.2, calculate the difference value of the two video frames, and determine the video frame when it is greater than threshold Th2. The difference value in this application is defined as: if the gray value difference E p of a certain pixel point is greater than threshold G T , then the bit position of this pixel point is 1, otherwise the bit position is 0, and then the sum of the bit values of all pixel points is calculated, denoted as S p . The formula is as follows:

[0064] E p = |g n -g n-1 |

[0065]

[0066] S p =∑B p

[0067] Sub-step 4.3, calculate the number of tracked video frames, and determine the video frame when it is equal to threshold Th3. The number of tracked video frames is defined as N, when the detection result is returned to the end, it starts to accumulate from 0, and N+1 every time a frame is tracked.

[0068] Sub-step 4.4, calculate the number of video frames that have been sent for detection but have not returned detection results, and stop sending when it is greater than threshold Th4. The number of video frames in transmission is defined as M, M+1 every time the adaptive detection module sends a frame to the cloud, and M-1 every time a group of detection results is returned to the end.

[0069] Sub-step 4.5, since the distance between the skin base stations will not be too close, this application considers that the skin base stations with coordinate error within 3m are the same skin base station. The absolute coordinates of the same skin base station in different video frames are different, and the weighted average of the absolute coordinates of the skin base station in the nearest 10 frames is taken as the final coordinate C f . The formula is as follows:

[0070]

[0071] As the whole device moves in the scene, the camera will gradually approach and then move away from the skin base station. The depth map obtained by depth estimation will be more and more accurate, so the weight of the newly obtained video frame is greater than that of the old video frame in the time sequence. In the above formula, the weight w i is set as:

[0072]

[0073] With reference to Figure 3 , an indoor skin base station detection and positioning device based on end-cloud cooperation comprises the following modules:

[0074] An image acquisition module is configured to continuously capture images of the indoor skin base station during the movement of the device.

[0075] A depth estimation module is configured to perform depth estimation on the images to obtain a depth map.

[0076] A video frame buffer module is configured to store the captured video frame sequence.

[0077] An adaptive submission module is configured to automatically determine whether the video frames meet the submission conditions and submit the frames meeting the conditions to the cloud side for detection.

[0078] In an embodiment, the adaptive submission module comprises:

[0079] A first calculation module is configured to calculate the cumulative tracking error D std of each video frame.

[0080] A first determination module is configured to determine the video frame whose D std is greater than a threshold Th1.

[0081] A second calculation module is configured to calculate the difference value S p of the current video frame and the previous video frame.

[0082] A second determination module is configured to determine the video frame whose S p is greater than a threshold Th2.

[0083] A third calculation module is configured to calculate the number M of the currently tracked video frames.

[0084] A third determination module is configured to determine the video frame whose N is equal to a threshold Th3.

[0085] A submission limiting module is configured to calculate the number M of the video frames currently submitted for detection but without returned results, and limit the submission of the video frames if the number is greater than a threshold Th4.

[0086] In an embodiment, the device further comprises:

[0087] a frame uploading module, configured to send the first frame when the device is started and the next frame of the video frame determined by the first determination module, the second determination module and the third determination module in the adaptive inspection module to the deep learning model through webRTC for detection;

[0088] an AI inference module, configured to perform target detection and feature extraction on the image transmitted by the inspection module, and transmit the label, feature point and bounding box of the skin base station to the tracking module on the terminal side through the websocket protocol;

[0089] a tracking module, configured to track the skin base station according to the feature point of the skin base station;

[0090] a coordinate output module, configured to finally output the absolute coordinates of the skin base station. First, the relative coordinates of the skin base station in the picture are determined according to the bounding box of the skin base station and the depth map of the current frame, and then the relative coordinates are converted into absolute coordinates, and then a plurality of absolute coordinates are weighted and averaged as the final result.

[0091] a result rendering module, configured to frame the skin base station in the picture in a prominent manner and display the label and the final coordinates above the skin base station.

[0092] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for indoor skin base station detection and positioning based on end-cloud cooperation, characterized in that, The method comprises the following steps: Collecting a skin base station image, performing depth estimation on the skin base station image to obtain a depth map; The cloud side obtains the bounding box, target label and target feature point of the skin base station image and sends them to the end side for target tracking, calculates the tracking cumulative error, frame difference value and the number of tracked video frames in the tracking process, sets the submission condition and respectively judges whether the submission condition is met, if yes, the next frame image is sent to the cloud side for information acquisition, KLT tracking is performed, if not, the final coordinates of the skin base station are calculated according to the depth map and the bounding box, and the skin base station positioning result is displayed; The depth estimation process comprises: performing gradient filtering on the skin base station image by using a Sobel operator, performing cost operation, replacing the cost value of each pixel point with the total sum of the cost values in the field, and obtaining the minimum optimal disparity of each pixel point after the replacement processing by using dynamic programming and conversion to obtain the depth map; The final coordinate calculation process comprises: calculating the relative coordinates of the skin base station according to the bounding box and the depth map, obtaining the absolute coordinates of the skin base station through coordinate system conversion, setting the absolute coordinate error preset value, if the error does not exceed the preset value, the skin base station is considered to be the same, and the weighted average result of the absolute coordinates obtained from the continuous multiple frames of images is taken as the final coordinates of the skin base station.

2. The indoor skin base station detection and positioning method based on end-cloud cooperation according to claim 1, wherein The process of collecting the skin base station image comprises: fixing a binocular camera on a four-wheel car, and moving the four-wheel car at a constant speed to collect the skin base station image.

3. The indoor skin base station detection and positioning method based on end-cloud cooperation according to claim 1, wherein The process of target tracking comprises: constructing a target detection model based on a YOLOv5s network, adding a 160 160 detecting a feature map, performing upsampling on the feature map after 17 layers in the head of the model, adding a small target detection layer to the detection layer, and performing target tracking by using the added model.

4. The indoor picocell detection and localization method based on end-cloud collaboration according to claim 1, characterized in that, The standard deviation of the moving distance of all feature points of the skin base station between the current frame and the previous frame is taken as the tracking cumulative error, and the gray value difference of the corresponding pixel points between the current frame and the previous frame is taken as the frame difference value; The number of tracked video frames is counted, and the number is increased by one for each tracked frame.

5. An indoor skin base station detection and positioning device based on end-cloud cooperation, used to implement the method of claim 1, characterized in that, It comprises: An end side platform composed of embedded computing devices; A cloud side platform composed of cloud servers; The end side platform comprises: An image collection module composed of a binocular camera and a four-wheel intelligent car, used for collecting images of indoor skin base stations; A depth estimation module used for depth estimation of images to obtain depth information; A video frame buffer module used for storing collected video frame sequences; An adaptive submission module used for adaptively submitting qualified video frames to the cloud side platform for detection according to the tracking error, the difference between the current frame and the previous frame, and the number of tracked video frames during tracking, thereby ensuring the accuracy of the skin base station coordinates while reducing the network bandwidth occupation caused by frame-by-frame detection; A frame uploading module used for uploading the first frame when the device starts and the next frame of the video frame determined by the first determination module, the second determination module and the third determination module in the adaptive submission module into a deep learning model for detection; A tracking module used for tracking the skin base station according to the feature points of the skin base station; A coordinate output module used for converting the relative coordinates of the skin base station in the picture into absolute coordinates, and then weighting and averaging multiple absolute coordinates as the final result. A result rendering module is configured to highlight the skin base station and its final coordinates in the picture. 6.The indoor skin base station detection and positioning device based on end-cloud cooperation according to claim 5, wherein, The adaptive submission module comprises: A first calculation module is configured to calculate the accumulated tracking error of each video frame. A first determination module is configured to determine the video frame with the accumulated tracking error greater than a threshold value. A second calculation module is configured to calculate the difference value between the current video frame and the previous video frame. A second determination module is configured to determine the video frame with the difference value greater than a threshold value. A third calculation module is configured to calculate the number of currently tracked video frames. A third determination module is configured to determine the video frame with the number of tracked video frames equal to a threshold value. A submission limiting module is configured to calculate the number of video frames currently submitted for detection but without returned results, and limit the submission of video frames if the number is greater than a threshold value. 7.The indoor skin base station detection and positioning device based on end-cloud cooperation according to claim 5, wherein, The cloud-side platform comprises: An AI inference module is configured to perform target detection and feature extraction on the image transmitted by the frame uploading module, and transmit the label, feature point and bounding box of the skin base station to the tracking module.

Images