Target positioning method for mobile terminal

Abstract

The invention provides a target positioning method for a mobile terminal, which comprises the steps that an initial image is acquired through a first device of the mobile terminal and is corrected, the initial image comprises a plurality of target objects, the acquired position coordinates of the initial image are initial geographic coordinates, and the first device comprises a first parameter; inputting the corrected initial image into a pre-trained first model to obtain a frame coordinate, a category and a confidence score of each target object; inputting the corrected initial image into a pre-trained second model to obtain an initial depth value matrix with the same length and width as the initial image; based on the frame coordinate of each target object, judging the overlapping degree of each target object; based on the overlapping degree of each target object, obtaining a predicted distance between each target object and the initial geographic coordinate; and according to the predicted distance, the first parameter of the first device and the initial geographic coordinate, calculating to obtain the geographic coordinate of each target object.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to a target positioning method for a mobile terminal. Background Art

[0002] Nowadays, modern cities are composed of countless complex elements, and these elements are changing rapidly. For example, various scenes including streets, parks, and squares are constantly changing dynamically (vehicles and pedestrians are constantly moving). Real-time acquisition of the location information of these objects is of great significance to life and other applications, such as traffic condition analysis, geographic situation analysis, etc.

[0003] Ideally, the distance can be measured by the propagation time between the GPS or Beidou signal and the receiver, and the position of the object on the map can be obtained by combining the principle of triangulation. However, in some scenarios, satellite signals cannot be covered or the target does not carry a receiver, so a less restricted solution is needed to locate the target. To solve these problems, the location of the target can be plotted on the map by measuring the distance and angle of the target and combining the longitude and latitude of the observation point. Usually, external devices such as lidar, ultrasonic sensors, and visual sensors can be used for ranging. However, these devices are expensive and easily affected by environmental interference. Summary of the invention

[0004] The present invention provides a target positioning method for a mobile terminal, comprising: acquiring an initial image through a first device of the mobile terminal and performing a correction process, wherein the initial image includes multiple target objects, the acquisition position coordinates of the initial image are initial geographic coordinates, and the first device includes a first parameter; inputting the corrected initial image into a pre-trained first model to obtain the frame coordinates, category and confidence score of each target object; inputting the corrected initial image into a pre-trained second model to obtain an initial depth value matrix with the same length and width as the initial image; based on the frame coordinates of each target object, determining the overlap degree of each target object; based on the overlap degree of each target object, obtaining the predicted distance between each target object and the initial geographic coordinates; and calculating the geographic coordinates of each target object according to the predicted distance, the first parameter of the first device and the initial geographic coordinates.

[0005] In the above scheme, the first device also includes a second parameter, and the initial image is obtained and corrected through the first device on the mobile terminal, and further includes: correcting the initial image based on the first parameter and the second parameter of the first device.

[0006] In the above scheme, the pre-trained first model and the pre-trained second model are both deployed on the first device, and the deployment process includes: according to the training data set, the preset first model and the second model are respectively trained to obtain the pre-trained first model and the pre-trained second model; the pre-trained first model and the pre-trained second model are respectively parameter-adjusted and model-converted to obtain the converted first model and the converted second model; the RKNN tool is used to convert the converted first model and the converted second model into RKNN models, and deployed on the first device.

[0007] In the above scheme, the degree of overlap of each target object is determined based on the bounding box coordinates of each target object, including: obtaining the center coordinates of each target object based on the bounding box coordinates of each target object; when the center coordinates of any target object are located within the bounding box of another target object, it is determined that there is overlap, wherein the target object with overlap is the first target object, and the target object without overlap is the second target object.

[0008] In the above scheme, based on the degree of overlap of each target object, the predicted distance between each target object and the initial geographic coordinates is obtained, including: when the target object is a second target object, according to the center coordinates of the second target object, the corresponding depth value is extracted from the initial depth value matrix to obtain the predicted distance between the second target object and the initial geographic coordinates.

[0009] In the above scheme, based on the degree of overlap of each target object, the predicted distance between each target object and the initial geographic coordinates is obtained, including: when the target object is a first target object, calculating the border difference set of other first target objects overlapping with the first target object, the difference set including the first difference set and the second difference set; based on the first difference set and the second difference set, determining the predicted distance between the first target object and the initial geographic coordinates.

[0010] In the above scheme, based on the first difference set and the second difference set, the predicted distance between the first target object and the initial geographic coordinates is determined, including: based on the first difference set and the second difference set, the corresponding depth value mean is calculated; when the depth value mean is greater than a preset threshold, the distance is represented by the median of the depth value corresponding to the larger difference set between the first difference set and the second difference set; when the depth value mean is less than the preset threshold, according to the center coordinates of the first target object, the corresponding depth value is extracted from the initial depth value matrix to determine the predicted distance between the first target object and the initial geographic coordinates.

[0011] In the above scheme, the geographic coordinates of each target object are calculated according to the predicted distance, the first parameter of the first device and the initial geographic coordinates, including: using satellite positioning or triangulation positioning to obtain the angle between the acquisition direction of the initial image and the earth's latitude; based on the angle, the center coordinates of the target object, the first parameter of the first device and the initial geographic coordinates, the actual width and height of the center of the target object from the center of the first device's viewing angle are calculated: based on the actual width and height, the distance between the acquisition position and the plane where the target object is located is obtained; based on the distance between the acquisition position and the plane where the target object is located, the corresponding geographic coordinates of the target object are obtained.

[0012] In the above scheme, the parameters of the pre-trained first model and the pre-trained second model are fine-tuned respectively, including: fine-tuning all parameters of the first model and fine-tuning some parameters of the second model.

[0013] In the above scheme, model conversion is performed on the pre-trained first model and the pre-trained second model respectively, and also includes: presetting standardized parameters of the image.

[0014] The technical solution of the embodiment of the present invention has at least the following beneficial effects:

[0015] (1) Based on the deep learning model, the intelligent algorithm of target detection and depth estimation is used to analyze the image, and the center coordinates of the target object frame are used to determine the overlap of the target. Different methods are used to obtain the distance of the target object in different situations to determine the location of the target object on the map. This method does not rely on additional equipment and only needs to process the pictures taken by the camera to obtain the results. It has low cost, short processing time, and few restrictions.

[0016] (2) In order to efficiently run the target detection and depth estimation algorithms on mobile devices, the present invention implements model compression and conversion and deploys it to mobile devices. After the model is lightweight, it can make full use of the terminal resources and improve the reasoning and operation efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a target positioning method for a mobile terminal according to an embodiment of the present invention is schematically shown;

[0018] Figure 2 An initial image diagram according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0020] First, the technical terms described in this article are explained and described as follows.

[0021] The YOLO-V5 model can adaptively scale the size of the image to adapt to the detection of targets of different scales. The backbone network of the model is composed of a lightweight ResNet. The neck is responsible for multi-scale feature fusion of the features extracted by the backbone and passing the features to the prediction layer. The head is mainly responsible for the final regression prediction to detect the location and category of the target. After the prediction is completed, non-maximum suppression (NMS) is used for post-processing to remove duplicate borders.

[0022] The DepthAnything-V2 model uses DINOv2 as an encoder for feature extraction and DPT as a decoder for depth regression. It can be pre-trained with a large amount of data and has strong generalization capabilities. Based only on high-quality synthetic images, it can be trained as a reliable teacher model based on DINOv2-G; accurate pseudo-depth labels can be generated on large-scale unlabeled real images; using pseudo-labeled real images to train the final student model can achieve robust generalization.

[0023] Figure 1 The flowchart of the target positioning method for a mobile terminal according to an embodiment of the present invention is schematically shown.

[0024] Please refer to Figure 1 The specific process of the mobile terminal-oriented target positioning method of the embodiment of the present invention includes operations S110~S160.

[0025] In operation S110, an initial image is acquired and rectified through a first device at a mobile terminal, wherein the initial image includes a plurality of target objects, the acquisition position coordinates of the initial image are initial geographic coordinates, and the first device includes first parameters.

[0026] In an embodiment of the present invention, the first device also includes a second parameter, and obtaining the initial image through the first device on the mobile terminal and performing correction processing also includes: correcting the initial image based on the first parameter and the second parameter of the first device.

[0027] Exemplarily, the first device may be a mobile device such as an image acquisition device camera, and the camera device includes multiple parameters, such as the first parameter is the intrinsic parameter matrix of the camera, the second parameter is the distortion coefficient, and other parameters also include rotation matrix and translation vector.

[0028] For the process of determining the camera parameters, for example, we can first take a black and white chessboard image from multiple angles, use OpenCV's cv2.findChessboardCorners() to find the foot points of the chessboard in the image, and obtain the three-dimensional points used for calibration and the corresponding two-dimensional point pairs on the image; then use cv2.calibrateCamera() for calibration, and return the calibration results, the camera's intrinsic parameter matrix, distortion coefficients, rotation matrix, and translation vector. The camera's intrinsic parameter matrix is:

[0029]

[0030] Among them, f x and f y is the focal length in pixels on the x and y axes, C x and C y are the principal point coordinates. In addition, the distortion coefficient matrix contains 4, 5 or 8 elements, which is determined by the characteristics of the camera lens.

[0031] Figure 2 An initial image diagram according to an embodiment of the present invention is schematically shown.

[0032] The initial image is obtained by shooting with a camera, and the initial image contains multiple target objects, for example, the target object may be a person in the image, a related object, or a pixel in the image. Figure 2 As shown, there are multiple target objects in the initial image, namely person 1, person 2 and bus 3. The position coordinates of the camera when taking the initial image are the initial physical coordinates, that is, the initial physical coordinates are the geographic coordinates of the shooting point on the map. Then, according to the intrinsic parameter matrix and distortion coefficient of the camera, the initial image is corrected so that the corrected initial image meets the subsequent first model and the input requirements of the first model.

[0033] In operation S120, the rectified initial image is input into a pre-trained first model to obtain the bounding box coordinates, category, and confidence score of each target object.

[0034] In operation S130, the corrected initial image is input into a pre-trained second model to obtain an initial depth value matrix having the same length and width as the initial image.

[0035] In an embodiment of the present invention, the above-mentioned corrected initial image is respectively input into a pre-trained first model and a pre-trained second model. In this embodiment, the first model can be a target detection model based on deep learning such as a YOLO-V5 model, and the second model can be a depth estimation model such as a DepthAnything-V2 model.

[0036] It should be noted that the pre-trained first model and the pre-trained second model are both deployed on the first device, and the deployment process includes: according to the training data set, the preset first model and the second model are trained respectively to obtain the pre-trained first model and the pre-trained second model; the pre-trained first model and the pre-trained second model are parameter-adjusted and model-converted respectively to obtain the converted first model and the converted second model; the converted first model and the converted second model are converted into RKNN models using the RKNN tool, and deployed on the first device. Among them, the model conversion of the pre-trained first model and the pre-trained second model also includes: the standardized parameters of the preset image.

[0037] Exemplarily, a YOLO-V5 model and a DepthAnything-V2 model are first constructed using a deep learning framework, and then the YOLO-V5 model and the DepthAnything-V2 model are pre-trained according to a training data set to obtain a pre-trained YOLO-V5 model and a pre-trained DepthAnything-V2 model.

[0038] Furthermore, the parameters of the pre-trained first model and the pre-trained second model are fine-tuned respectively, including: fine-tuning all parameters of the first model and fine-tuning some parameters of the second model.

[0039] For example, for the YOLO-V5 model, full parameter fine-tuning is used, the learning rate is 0.00001, and training is performed for 100 epochs. For the DepthAnything-V2 model, partial parameter fine-tuning is used, only the bias parameter is fine-tuned, and the rest of the parameters are frozen, the learning rate is 0.001, and training is performed for 100 epochs.

[0040] For another example, you can also choose different fine-tuning methods based on the amount of data in the training set and the characteristics of the application scenario, such as prompt learning, adding an adaptor, and only fine-tuning the last few layers.

[0041] In addition, in an embodiment of the present invention, the model is also optimized, and a joint optimization strategy can be used during model optimization. That is, a new connection module is added between the YOLO-V5 and DepthAnything-V2 models, which can be feature splicing, cross attention or semantic alignment, so that the intermediate features of the two are fused to form mutual guidance and promote learning more knowledge.

[0042] Furthermore, the trained model is converted into a script file using a preset function, for example, the torch.jit.trace function is used to convert the trained model into a TorchScript script file. At the same time, the image standardization parameters are preset, for example, the mean of the three RGB channels is set to [123.675, 116.28, 103.53], and the standard deviation is set to [58.395, 58.395, 58.395]. Then, the RKNN tool is used to convert the TorchScript model (.pt file) into an RKNN model (.RKNN file). The RKNN model is more lightweight (data type is float16), has higher running efficiency, and can run on the NPU for inference. Finally, deploy the RKNN model on the camera, that is, install the operating system (Ubuntu or Debian), RKNN-ToolKIT-LITE toolkit, and RKNN SERVER tool on the RK3588 board, send the RKNN model to the RK3588 board, and start the RKNN SERVER service on the board.

[0043] Through the embodiments of the present invention, compared with the current deep learning-based methods, which all use general frameworks to build models, these commonly used frameworks have low operating efficiency. When deployed on mobile terminals, they will occupy a large amount of resources and require a long reasoning time, making them difficult to apply in practice. The embodiments of the present invention convert the deep learning model into the .rknn format, reduce the complexity of the operation, and can quickly complete the positioning analysis of the target object in the image.

[0044] The rectified initial image is input into the pre-trained YOLO-V5 model to obtain the bounding box coordinates (for example, the upper left corner is (X1, Y1), the lower right corner is (X2, Y2)), category and confidence score of each target object in the initial image. The rectified initial image is input into the pre-trained DepthAnything-V2 model to obtain an initial depth value matrix with the same length and width as the initial image. Each element in the initial depth value matrix represents the absolute distance between each target object in the initial image and the camera shooting point, that is, the depth value (in meters).

[0045] In operation S140 , the overlapping degree of each target object is determined based on the bounding box coordinates of each target object.

[0046] In an embodiment of the present invention, the degree of overlap of each target object is determined based on the bounding box coordinates of each target object, including: obtaining the center coordinates of each target object based on the bounding box coordinates of each target object; when the center coordinates of any target object are within the bounding box of another target object, it is determined that there is overlap, wherein the target object with overlap is the first target object, and the target object without overlap is the second target object.

[0047] According to the YOLO-V5 model output, the bounding box coordinates of each target object are calculated to obtain the center coordinates of each target object. , When the center coordinates of one target object are located within the border of another target object, it is considered that there is an overlap. The target object with the overlap is the first target object, and the target object without the overlap is the second target object.

[0048] In operation S150 , a predicted distance between each target object and the initial geographic coordinates is obtained based on the overlap degree of each target object.

[0049] In an embodiment of the present invention, based on the degree of overlap of each target object, a predicted distance between each target object and the initial geographic coordinates is obtained, including: when the target object is a second target object, according to the center coordinates of the second target object, the corresponding depth value is extracted from the initial depth value matrix to obtain the predicted distance between the second target object and the initial geographic coordinates.

[0050] Specifically, when there is no overlap with the target object, that is, when the target object is the second target object, the corresponding depth value is extracted from the initial depth value matrix output by DepthAnything-V2 according to the center coordinates of the second target object as the predicted distance between the second target object and the initial geographic coordinates.

[0051] In an embodiment of the present invention, based on the degree of overlap of each target object, a predicted distance between each target object and the initial geographic coordinates is obtained, and it also includes: when the target object is a first target object, calculating the border difference set of other first target objects overlapping with the first target object, the difference set includes a first difference set and a second difference set; based on the first difference set and the second difference set, determining the predicted distance between the first target object and the initial geographic coordinates.

[0052] Specifically, when the target objects overlap, that is, the first target objects, the border difference sets of other first target objects overlapping with the first target object are calculated. For example, the areas within the borders of the two first target objects in the initial image are represented by a and b respectively, and the difference sets between the two are respectively the first difference sets D a =ab, the second difference set D b =ba.

[0053] Furthermore, based on the first difference set and the second difference set, the predicted distance between the first target object and the initial geographic coordinates is determined, including: based on the first difference set and the second difference set, calculating the corresponding mean depth value; when the mean depth value is greater than a preset threshold, using the median of the depth value corresponding to the larger difference set between the first difference set and the second difference set to represent the distance; when the mean depth value is less than the preset threshold, according to the center coordinates of the first target object, extracting the corresponding depth value from the initial depth value matrix to determine the predicted distance between the first target object and the initial geographic coordinates.

[0054] Specifically, according to the first difference set D a =ab, the second difference set D b = ba, calculate D a and D b The corresponding depth value mean, when the mean is less than the preset threshold, that is, the mean is small, it means that the distance between the first target object and the camera shooting point is close, and the first target object is not blocked. At this time, the corresponding depth value can still be extracted from the initial depth value matrix according to the center coordinates of the first target object as the predicted distance between the first target object and the initial geographic coordinates. When the mean is greater than the preset threshold, that is, the mean is large, it means that the distance between the first target object and the camera shooting point is far, and the first target object is blocked by an object, then the median of the depth value corresponding to the larger difference set of the first difference set and the second difference set is used to represent the predicted distance between the first target object and the initial geographic coordinates.

[0055] In operation S160 , the geographic coordinates of each target object are calculated according to the predicted distance, the first parameter of the first device, and the initial geographic coordinates.

[0056] In an embodiment of the present invention, the geographic coordinates of each target object are calculated based on the predicted distance, the first parameter of the first device and the initial geographic coordinates, including: using satellite positioning or triangulation positioning to obtain the angle between the acquisition direction of the initial image and the earth's latitude; based on the angle, the center coordinates of the target object, the first parameter of the first device and the initial geographic coordinates, calculating the actual width and height of the center of the target object from the center of the viewing angle of the first device: based on the actual width and height, obtaining the distance between the acquisition position and the plane where the target object is located; based on the distance between the acquisition position and the plane where the target object is located, obtaining the corresponding geographic coordinates of the target object.

[0057] It can be understood that the geographic coordinates of the target object on the map are calculated through the geographic coordinates of the shooting point on the map, the shooting direction, the parameters of the camera, and the predicted distance between the target object and the initial geographic coordinates.

[0058] For example, the geographical coordinates of the shooting point on the map are (M x ,M y), the angle between the shooting direction and the earth's latitude is (For example, using satellite positioning or triangulation positioning), the coordinates of the center of the target object in the image are ( x ,O y ), the distance from the shooting point is D (that is, the depth value), the internal parameter matrix of the camera is M, and the actual width and height of the center of the target object from the center of the camera's viewing angle are calculated:

[0059]

[0060] The distance between the shooting point and the plane where the target object is located is:

[0061]

[0062] Furthermore, the geographic coordinates of the target object on the map can be obtained as:

[0063]

[0064] The embodiments of the present invention solve the problems that the current satellite positioning-based methods in the technology of obtaining the geographic location of an object require satellite signal coverage, and the target needs to carry a signal receiving device, and the hardware sensor-based methods are high in cost, long in processing time, and are easily affected by the environment. The embodiments of the present invention are based on a deep learning model, using intelligent algorithms for target detection and monocular depth estimation to analyze images, use the center coordinates of the target object frame to determine the overlap of the target, and use different methods to obtain the distance of the target object for different situations to determine the position of the target object on the map. This embodiment does not rely on additional equipment and devices, and only needs to process the pictures taken by the camera to obtain the results, with low cost, short processing time, and few restrictions.

[0065] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A target positioning method for a mobile terminal, comprising: Acquire an initial image through a first device on a mobile terminal and perform correction processing, wherein the initial image includes a plurality of target objects, the acquisition position coordinates of the initial image are initial geographic coordinates, and the first device includes a first parameter; Inputting the corrected initial image into a pre-trained first model to obtain the bounding box coordinates, category and confidence score of each target object; Input the corrected initial image into a pre-trained second model to obtain an initial depth value matrix with the same length and width as the initial image; Based on the frame coordinates of each target object, determining the degree of overlap of each target object; Based on the overlap degree of each target object, a predicted distance between each target object and the initial geographic coordinates is obtained; The geographic coordinates of each of the target objects are calculated according to the predicted distance, the first parameter of the first device and the initial geographic coordinates.

2. According to the mobile terminal-oriented target positioning method of claim 1, the first device further includes a second parameter, and the obtaining of the initial image by the first device of the mobile terminal and performing correction processing further includes: Based on the first parameter and the second parameter of the first device, the initial image is rectified.

3. According to the mobile terminal-oriented target positioning method according to claim 1 or 2, the pre-trained first model and the pre-trained second model are both deployed on the first device, and the deployment process includes: According to the training data set, the preset first model and the second model are trained respectively to obtain a pre-trained first model and a pre-trained second model; Fine-tune parameters and perform model conversion on the pre-trained first model and the pre-trained second model to obtain a converted first model and a converted second model; The converted first model and the converted second model are converted into RKNN models using an RKNN tool, and deployed on the first device.

4. The mobile terminal-oriented target positioning method according to claim 1, wherein the step of determining the degree of overlap of each target object based on the frame coordinates of each target object comprises: Based on the frame coordinates of each target object, the center coordinates of each target object are obtained; When the central coordinates of any of the target objects are located within the frame of another target object, it is determined that there is overlap, wherein the target object with overlap is the first target object, and the target object without overlap is the second target object.

5. According to the mobile terminal-oriented target positioning method of claim 1 or 4, obtaining the predicted distance between each target object and the initial geographic coordinates based on the overlap degree of each target object comprises: When the target object is a second target object, a corresponding depth value is extracted from the initial depth value matrix according to the center coordinates of the second target object to obtain a predicted distance between the second target object and the initial geographic coordinates.

6. According to the mobile terminal-oriented target positioning method of claim 1 or 4, obtaining the predicted distance between each target object and the initial geographic coordinates based on the overlap degree of each target object comprises: When the target object is a first target object, calculating a border difference set of other first target objects overlapping with the first target object, the difference set including a first difference set and a second difference set; Based on the first difference set and the second difference set, a predicted distance between the first target object and the initial geographic coordinates is determined.

7. The mobile terminal-oriented target positioning method according to claim 6, wherein determining the predicted distance between the first target object and the initial geographic coordinates based on the first difference set and the second difference set comprises: Calculating corresponding depth value means based on the first difference set and the second difference set; When the mean of the depth values ​​is greater than a preset threshold, the median of the depth values ​​corresponding to the larger difference set between the first difference set and the second difference set is used to represent the distance; When the mean depth value is less than a preset threshold, a corresponding depth value is extracted from the initial depth value matrix according to the center coordinates of the first target object to determine a predicted distance between the first target object and the initial geographic coordinates.

8. The mobile terminal-oriented target positioning method according to claim 4, wherein the step of calculating the geographic coordinates of each target object according to the predicted distance, the first parameter of the first device and the initial geographic coordinates comprises: Obtaining the angle between the acquisition direction of the initial image and the earth's latitude using satellite positioning or triangulation positioning; Based on the angle, the center coordinates of the target object, the first parameter of the first device and the initial geographic coordinates, the actual width and height of the center of the target object from the center of the viewing angle of the first device are calculated: Based on the actual width and height, obtaining the distance between the acquisition position and the plane where the target object is located; Based on the distance between the acquisition position and the plane where the target object is located, the corresponding geographic coordinates of the target object are obtained.

9. According to the mobile terminal-oriented target positioning method of claim 3, the step of fine-tuning parameters of the pre-trained first model and the pre-trained second model respectively comprises: All parameters of the first model are fine-tuned, and some parameters of the second model are fine-tuned.

10. The mobile terminal-oriented target positioning method according to claim 3, wherein the pre-trained first model and the pre-trained second model are respectively converted into models, further comprising: Standardized parameters for preset images.

Images