A target positioning method for mobile terminals

Abstract

The present invention provides a target positioning method for a mobile terminal, comprising: acquiring an initial image through a first device on the mobile terminal and performing correction processing, wherein the initial image includes multiple target objects, the acquisition position coordinates of the initial image are initial geographic coordinates, and the first device includes first parameters; inputting the corrected initial image into a pre-trained first model to obtain the frame coordinates, category and confidence score of each target object; inputting the corrected initial image into a pre-trained second model to obtain an initial depth value matrix with the same length and width as the initial image; based on the frame coordinates of each target object, determining the overlap degree of each target object; based on the overlap degree of each target object, obtaining a predicted distance between each target object and the initial geographic coordinates; and calculating the geographic coordinates of each target object according to the predicted distance, the first parameter of the first device and the initial geographic coordinates.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a target positioning method for a mobile terminal. Background Art

[0002] Modern cities are composed of countless complex elements that are constantly changing. For example, streets, parks, and squares are constantly changing (vehicles and pedestrians are constantly moving). Real-time location information for these objects is crucial for everyday life and other applications, such as traffic condition analysis and geographic situation analysis.

[0003] Ideally, the distance between the GPS or Beidou signal and the receiver can be measured using the propagation time between them, and the object's position on the map can be determined using triangulation. However, in some scenarios, satellite signals may not be able to reach the target or the target does not carry a receiver, so a less restrictive solution is needed to locate the target. To address these issues, the target's position can be plotted on a map by measuring its distance and angle, combined with the longitude and latitude of the observation point. Distance measurement is typically performed using external devices such as lidar, ultrasonic sensors, and visual sensors. However, these devices are expensive and susceptible to environmental interference. Summary of the Invention

[0004] The present invention provides a target positioning method for a mobile terminal, comprising: acquiring an initial image through a first device of the mobile terminal and performing correction processing, wherein the initial image includes multiple target objects, the acquisition position coordinates of the initial image are initial geographic coordinates, and the first device includes first parameters; inputting the corrected initial image into a pre-trained first model to obtain the frame coordinates, category and confidence score of each target object; inputting the corrected initial image into a pre-trained second model to obtain an initial depth value matrix with the same length and width as the initial image; based on the frame coordinates of each target object, determining the degree of overlap of each target object; based on the degree of overlap of each target object, obtaining a predicted distance between each target object and the initial geographic coordinates; and calculating the geographic coordinates of each target object according to the predicted distance, the first parameter of the first device and the initial geographic coordinates.

[0005] In the above solution, the first device also includes a second parameter, and the initial image is obtained and corrected through the first device on the mobile terminal, and further includes: correcting the initial image based on the first parameter and the second parameter of the first device.

[0006] In the above scheme, the pre-trained first model and the pre-trained second model are both deployed on the first device, and the deployment process includes: according to the training data set, the preset first model and the second model are trained respectively to obtain the pre-trained first model and the pre-trained second model; the pre-trained first model and the pre-trained second model are parameter fine-tuned and the model is converted respectively to obtain the converted first model and the converted second model; the RKNN tool is used to convert the converted first model and the converted second model into RKNN models, and deployed on the first device.

[0007] In the above scheme, the degree of overlap of each target object is determined based on the bounding box coordinates of each target object, including: calculating the center coordinates of each target object based on the bounding box coordinates of each target object; when the center coordinates of any target object are within the bounding box of another target object, it is determined that there is overlap, wherein the target object with overlap is the first target object, and the target object without overlap is the second target object.

[0008] In the above scheme, based on the degree of overlap of each target object, the predicted distance between each target object and the initial geographic coordinates is obtained, including: when the target object is the second target object, according to the center coordinates of the second target object, the corresponding depth value is extracted from the initial depth value matrix to obtain the predicted distance between the second target object and the initial geographic coordinates.

[0009] In the above scheme, based on the degree of overlap of each target object, the predicted distance between each target object and the initial geographic coordinates is obtained, including: when the target object is a first target object, calculating the border difference set of other first target objects overlapping with the first target object, the difference set including the first difference set and the second difference set; based on the first difference set and the second difference set, determining the predicted distance between the first target object and the initial geographic coordinates.

[0010] In the above scheme, based on the first difference set and the second difference set, the predicted distance between the first target object and the initial geographic coordinates is determined, including: based on the first difference set and the second difference set, calculating the corresponding depth value mean; when the depth value mean is greater than a preset threshold, using the median of the depth values ​​corresponding to the larger difference set between the first difference set and the second difference set to represent the distance; when the depth value mean is less than the preset threshold, according to the center coordinates of the first target object, extracting the corresponding depth value from the initial depth value matrix to determine the predicted distance between the first target object and the initial geographic coordinates.

[0011] In the above scheme, the geographic coordinates of each target object are calculated based on the predicted distance, the first parameter of the first device and the initial geographic coordinates, including: using satellite positioning or triangulation positioning to obtain the angle between the acquisition direction of the initial image and the earth's latitude; based on the angle, the center coordinates of the target object, the first parameter of the first device and the initial geographic coordinates, the actual width and height of the center of the target object from the center of the first device's field of view are calculated: based on the actual width and height, the distance between the acquisition position and the plane where the target object is located is obtained; based on the distance between the acquisition position and the plane where the target object is located, the corresponding geographic coordinates of the target object are obtained.

[0012] In the above solution, the parameters of the pre-trained first model and the pre-trained second model are fine-tuned respectively, including: fine-tuning all parameters of the first model and fine-tuning some parameters of the second model.

[0013] In the above solution, model conversion is performed on the pre-trained first model and the pre-trained second model respectively, and also includes: presetting the standardized parameters of the image.

[0014] The technical solution of the embodiment of the present invention has at least the following beneficial effects:

[0015] (1) Based on a deep learning model, the intelligent algorithms of target detection and depth estimation are used to analyze the image. The center coordinates of the target object frame are used to determine the overlap of the target. Different methods are used to obtain the distance of the target object in different situations to determine the location of the target object on the map. This method does not rely on additional equipment and can obtain results by simply processing the images taken by the camera. It is low-cost, has a short processing time, and has few restrictions.

[0016] (2) In order to efficiently run the target detection and depth estimation algorithms on mobile devices, the present invention implements model compression and conversion and deploys it to mobile devices. After the model is lightweight, it can make full use of the terminal resources and improve the reasoning and operation efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The following schematically shows a flow chart of a mobile terminal-oriented target positioning method according to an embodiment of the present invention;

[0018] Figure 2 An initial image diagram according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0020] First, the technical terms described in this article are explained and described as follows.

[0021] The YOLO-V5 model can adaptively scale images to detect objects of different scales. The model's backbone network consists of a lightweight ResNet. The neck is responsible for multi-scale feature fusion of the features extracted by the backbone and passing the features to the prediction layer. The head is mainly responsible for the final regression prediction to detect the location and category of the object. After the prediction is completed, non-maximum suppression (NMS) is used for post-processing to remove duplicate borders.

[0022] The DepthAnything-V2 model uses DINOv2 as the encoder for feature extraction and DPT as the decoder for depth regression. It can be pre-trained using a large amount of data and has strong generalization capabilities. Based solely on high-quality synthetic images, it can be trained as a reliable teacher model based on DINOv2-G. It can generate accurate pseudo-depth labels on large-scale unlabeled real images. Using pseudo-labeled real images to train the final student model achieves robust generalization.

[0023] Figure 1 The flowchart of the mobile terminal-oriented target positioning method according to an embodiment of the present invention is schematically shown.

[0024] Please refer to Figure 1 The specific process of the mobile terminal-oriented target positioning method according to the embodiment of the present invention includes operations S110 to S160.

[0025] In operation S110, an initial image is acquired and rectified by a first device on a mobile terminal, wherein the initial image includes a plurality of target objects, the acquisition location coordinates of the initial image are initial geographic coordinates, and the first device includes first parameters.

[0026] In an embodiment of the present invention, the first device also includes a second parameter, and obtaining the initial image through the first device on the mobile terminal and performing correction processing also includes: correcting the initial image based on the first parameter and the second parameter of the first device.

[0027] Exemplarily, the first device can be a mobile device such as an image acquisition device camera. The camera device includes multiple parameters, such as the first parameter is the intrinsic parameter matrix of the camera, the second parameter is the distortion coefficient, and other parameters also include rotation matrix and translation vector.

[0028] For example, the camera parameters can be determined by first photographing a black and white chessboard from multiple angles and using OpenCV's cv2.findChessboardCorners() to find the corners of the chessboard in the image. This yields the three-dimensional points used for calibration and the corresponding two-dimensional point pairs on the image. Calibration is then performed using cv2.calibrateCamera(), which returns the calibration results, the camera's intrinsic parameter matrix, distortion coefficients, rotation matrix, and translation vector. The camera's intrinsic parameter matrix is:

[0029]

[0030] Among them, f x and f y is the focal length in pixels on the x and y axes, C x and C y are the principal point coordinates. In addition, the distortion coefficient matrix contains 4, 5, or 8 elements, which is determined by the characteristics of the camera lens.

[0031] Figure 2 An initial image diagram according to an embodiment of the present invention is schematically shown.

[0032] The initial image is captured by a camera, and the initial image contains multiple target objects, such as people, related objects, or pixels in the image. Figure 2 As shown, the initial image contains multiple target objects: Person 1, Person 2, and Bus 3. The camera's position coordinates when capturing the initial image are the initial physical coordinates, which are the geographic coordinates of the capture point on a map. The initial image is then rectified based on the camera's intrinsic parameter matrix and distortion coefficients to ensure that the rectified initial image meets the subsequent first model and the input requirements of the first model.

[0033] In operation S120 , the rectified initial image is input into a pre-trained first model to obtain the bounding box coordinates, category, and confidence score of each target object.

[0034] In operation S130 , the corrected initial image is input into a pre-trained second model to obtain an initial depth value matrix having the same length and width as the initial image.

[0035] In an embodiment of the present invention, the above-mentioned corrected initial image is input into a pre-trained first model and a pre-trained second model respectively. In this embodiment, the first model can be a target detection model based on deep learning such as the YOLO-V5 model, and the second model can be a depth estimation model such as the DepthAnything-V2 model.

[0036] It should be noted that the pre-trained first model and the pre-trained second model are both deployed on the first device. The deployment process includes: training the preset first model and the pre-trained second model according to the training data set to obtain the pre-trained first model and the pre-trained second model; fine-tuning the parameters of the pre-trained first model and the pre-trained second model and performing model conversion to obtain the converted first model and the converted second model; using the RKNN tool to convert the converted first model and the converted second model into RKNN models, and deploying them on the first device. Among them, performing model conversion on the pre-trained first model and the pre-trained second model also includes: presetting the standardized parameters of the image.

[0037] For example, a YOLO-V5 model and a DepthAnything-V2 model are first constructed using a deep learning framework, and then the YOLO-V5 model and the DepthAnything-V2 model are pre-trained according to a training data set to obtain a pre-trained YOLO-V5 model and a pre-trained DepthAnything-V2 model.

[0038] Furthermore, the parameters of the pre-trained first model and the pre-trained second model are fine-tuned respectively, including: fine-tuning all parameters of the first model and fine-tuning some parameters of the second model.

[0039] For example, for the YOLO-V5 model, full parameter fine-tuning is used with a learning rate of 0.00001 and training for 100 epochs. For the DepthAnything-V2 model, partial parameter fine-tuning is used, where only the bias parameter is fine-tuned while the remaining parameters are frozen with a learning rate of 0.001 and training for 100 epochs.

[0040] For another example, you can also choose different fine-tuning methods based on the amount of data in the training set and the characteristics of the application scenario, such as prompt learning, adding an adaptor, and fine-tuning only the last few layers.

[0041] In addition, in embodiments of the present invention, the model is also optimized using a joint optimization strategy. Specifically, a new connection module is added between the YOLO-V5 and DepthAnything-V2 models. This module can be feature concatenation, cross-attention, or semantic alignment. This allows the intermediate features of the two models to be fused, providing mutual guidance and promoting the learning of more knowledge.

[0042] Next, use a pre-set function to convert the trained model into a script file, such as the torch.jit.trace function. Also, preset image normalization parameters, such as setting the mean of the three RGB channels to [123.675, 116.28, 103.53] and the standard deviation to [58.395, 58.395, 58.395]. Then, use the RKNN tool to convert the TorchScript model (.pt file) into an RKNN model (.RKNN file). The RKNN model is more lightweight (data type is float16), more efficient, and can run on the NPU for inference. Finally, deploy the RKNN model on the camera. That is, install the operating system (Ubuntu or Debian), RKNN-ToolKIT-LITE toolkit, and RKNN SERVER tool on the RK3588 board. Then send the RKNN model to the RK3588 board and start the RKNN SERVER service on the board.

[0043] Compared to existing deep learning-based methods that typically use general frameworks to build models, these commonly used frameworks are inefficient, consume significant resources, and require long inference times when deployed on mobile devices, making them difficult to implement in practice. The embodiments of the present invention convert deep learning models into the .rknn format, reducing operational complexity and enabling rapid localization analysis of target objects in images.

[0044] The rectified initial image is fed into the pre-trained YOLO-V5 model to obtain the bounding box coordinates (for example, (X1, Y1) for the upper left corner and (X2, Y2) for the lower right corner), category, and confidence score for each target object in the initial image. The rectified initial image is fed into the pre-trained DepthAnything-V2 model to obtain an initial depth matrix with the same length and width as the initial image. Each element in the initial depth matrix represents the absolute distance between each target object in the initial image and the camera's capture point, i.e., the depth value (in meters).

[0045] In operation S140 , the degree of overlap of each target object is determined based on the bounding box coordinates of each target object.

[0046] In an embodiment of the present invention, the degree of overlap of each target object is determined based on the bounding box coordinates of each target object, including: obtaining the center coordinates of each target object based on the bounding box coordinates of each target object; when the center coordinates of any target object are within the bounding box of another target object, it is determined that there is overlap, wherein the target object with overlap is the first target object, and the target object without overlap is the second target object.

[0047] According to the YOLO-V5 model output, the bounding box coordinates of each target object are calculated to obtain the center coordinates of each target object. , When the center coordinates of one target object are within the bounding box of another target object, it is considered that there is an overlap. The target object with the overlap is the first target object, and the target object without the overlap is the second target object.

[0048] In operation S150 , a predicted distance between each target object and the initial geographic coordinates is obtained based on the overlap degree of each target object.

[0049] In an embodiment of the present invention, based on the degree of overlap of each target object, a predicted distance between each target object and the initial geographic coordinates is obtained, including: when the target object is a second target object, according to the center coordinates of the second target object, the corresponding depth value is extracted from the initial depth value matrix to obtain the predicted distance between the second target object and the initial geographic coordinates.

[0050] Specifically, when there is no overlap with the target object, that is, when the target object is the second target object, the corresponding depth value is extracted from the initial depth value matrix output by DepthAnything-V2 according to the center coordinates of the second target object as the predicted distance between the second target object and the initial geographic coordinates.

[0051] In an embodiment of the present invention, based on the degree of overlap of each target object, a predicted distance between each target object and the initial geographic coordinates is obtained, and the method further includes: when the target object is a first target object, calculating a border difference set of other first target objects overlapping with the first target object, the difference set including a first difference set and a second difference set; and determining a predicted distance between the first target object and the initial geographic coordinates based on the first difference set and the second difference set.

[0052] Specifically, when the target objects overlap, that is, the first target objects, the difference sets of the bounding boxes of the other first target objects overlapping with the first target object are calculated. For example, the areas within the bounding boxes of the two first target objects in the initial image are represented by a and b respectively, and the difference sets between the two are the first difference sets D a =ab, second difference set D b =ba.

[0053] Furthermore, based on the first difference set and the second difference set, the predicted distance between the first target object and the initial geographic coordinates is determined, including: calculating the corresponding depth value mean based on the first difference set and the second difference set; when the depth value mean is greater than a preset threshold, using the median of the depth values ​​corresponding to the larger difference set between the first difference set and the second difference set to represent the distance; when the depth value mean is less than the preset threshold, extracting the corresponding depth value from the initial depth value matrix according to the center coordinates of the first target object, and determining the predicted distance between the first target object and the initial geographic coordinates.

[0054] Specifically, according to the first difference set D a =ab, second difference set D b =ba, calculate D respectively a and D b The corresponding depth value mean, when the mean is less than the preset threshold, that is, the mean is small, it means that the distance between the first target object and the camera shooting point is relatively close, and the first target object is not blocked. At this time, the corresponding depth value can still be extracted from the initial depth value matrix based on the center coordinates of the first target object to serve as the predicted distance between the first target object and the initial geographic coordinates. When the mean is greater than the preset threshold, that is, the mean is large, it means that the distance between the first target object and the camera shooting point is relatively far, and the first target object is blocked by an object. The median of the depth values ​​corresponding to the larger difference set of the first difference set and the second difference set is used to represent the predicted distance between the first target object and the initial geographic coordinates.

[0055] In operation S160 , the geographic coordinates of each target object are calculated based on the predicted distance, the first parameter of the first device, and the initial geographic coordinates.

[0056] In an embodiment of the present invention, the geographic coordinates of each target object are calculated based on the predicted distance, the first parameter of the first device and the initial geographic coordinates, including: using satellite positioning or triangulation positioning to obtain the angle between the acquisition direction of the initial image and the earth's latitude; based on the angle, the center coordinates of the target object, the first parameter of the first device and the initial geographic coordinates, calculating the actual width and height of the center of the target object from the center of the viewing angle of the first device; based on the actual width and height, obtaining the distance between the acquisition position and the plane where the target object is located; based on the distance between the acquisition position and the plane where the target object is located, obtaining the corresponding geographic coordinates of the target object.

[0057] It is understandable that the geographic coordinates of the target object on the map are calculated using the geographic coordinates of the shooting point on the map, the shooting direction, the camera parameters, and the predicted distance between the target object and the initial geographic coordinates.

[0058] For example, the geographical coordinates of the shooting point on the map are (M x ,M y), the angle between the shooting direction and the earth's latitude is (For example, using satellite positioning or triangulation positioning), the coordinates of the center of the target object in the image are ( x ,O y ), the distance from the shooting point is D (also known as the depth value), the camera internal parameter matrix is ​​M, and the actual width and height of the center of the target object from the center of the camera perspective are calculated:

[0059]

[0060] The distance between the shooting point and the plane where the target object is located is:

[0061]

[0062] Furthermore, the geographic coordinates of the target object on the map can be obtained as:

[0063]

[0064] The embodiments of the present invention solve the problems in the current technology of obtaining the geographic location of an object, such as the method based on satellite positioning requiring satellite signal coverage and the target carrying a signal receiving device, and the method based on hardware sensors being high in cost, long in processing time, and susceptible to environmental influences. The embodiments of the present invention are based on a deep learning model, using an intelligent algorithm of target detection and monocular depth estimation to analyze images, using the center coordinates of the target object frame to determine the overlap of the target, and using different methods to obtain the distance of the target object for different situations to determine the position of the target object on the map. This embodiment does not rely on additional equipment and only needs to process the picture taken by the camera to obtain the result, with low cost, short processing time, and few restrictions.

[0065] The above specific embodiments further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target positioning method for a mobile terminal, comprising: Acquire an initial image through a first device on a mobile terminal and perform correction processing, wherein the initial image includes a plurality of target objects, the acquisition location coordinates of the initial image are initial geographic coordinates, and the first device includes first parameters; Inputting the corrected initial image into a pre-trained first model to obtain the bounding box coordinates, category, and confidence score of each target object; Inputting the corrected initial image into a pre-trained second model to obtain an initial depth value matrix with the same length and width as the initial image; Determining the degree of overlap of each target object based on the bounding box coordinates of each target object; Based on the degree of overlap of each target object, a predicted distance between each target object and the initial geographic coordinates is obtained; Calculate the geographic coordinates of each target object according to the predicted distance, the first parameter of the first device, and the initial geographic coordinates; Wherein, obtaining the predicted distance between each target object and the initial geographic coordinates based on the overlap degree of each target object includes: When the target is a first target object, calculating a difference set of a bounding box overlapping with the first target object; Calculating the corresponding depth value mean based on the border difference set; When the mean of the depth values ​​is greater than a preset threshold, the median of the depth values ​​corresponding to the bounding box difference set is used to represent the predicted distance; When the mean depth value is less than a preset threshold, the corresponding depth value is extracted from the initial depth value matrix according to the center coordinates of the first target object to determine the predicted distance between the first target object and the initial geographic coordinates.

2. The mobile-oriented target positioning method according to claim 1, wherein the first device further comprises a second parameter, and the obtaining of the initial image and performing correction processing by the first device on the mobile terminal further comprises: The initial image is rectified based on the first parameter and the second parameter of the first device.

3. The mobile-oriented target positioning method according to claim 1 or 2, wherein the pre-trained first model and the pre-trained second model are both deployed on the first device, and the deployment process includes: According to the training data set, the preset first model and the second model are trained respectively to obtain a pre-trained first model and a pre-trained second model; Performing parameter fine-tuning and model conversion on the pre-trained first model and the pre-trained second model, respectively, to obtain a converted first model and a converted second model; The converted first model and the converted second model are converted into RKNN models using an RKNN tool and deployed on the first device.

4. The mobile terminal-oriented target positioning method according to claim 1, wherein determining the degree of overlap of each target object based on the bounding box coordinates of each target object comprises: Calculating the center coordinates of each target object based on the frame coordinates of each target object; When the center coordinates of any target object are located within the frame of another target object, it is determined that there is overlap, wherein the target object with overlap is the first target object, and the target object without overlap is the second target object.

5. The mobile terminal-oriented target positioning method according to claim 1 or 4, wherein obtaining the predicted distance between each target object and the initial geographic coordinates based on the degree of overlap of each target object comprises: When the target object is a second target object, a corresponding depth value is extracted from the initial depth value matrix according to the center coordinates of the second target object to obtain a predicted distance between the second target object and the initial geographic coordinates.

6. The mobile terminal-oriented target positioning method according to claim 1 or 4, wherein obtaining the predicted distance between each target object and the initial geographic coordinates based on the degree of overlap of each target object comprises: When the target object is a first target object, calculating a border difference set of other first target objects overlapping with the first target object, the difference set including a first difference set and a second difference set; Based on the first difference set and the second difference set, a predicted distance between the first target object and the initial geographic coordinates is determined.

7. The mobile terminal-oriented target positioning method according to claim 4, wherein the calculating the geographic coordinates of each target object based on the predicted distance, the first parameter of the first device, and the initial geographic coordinates comprises: Obtaining the angle between the acquisition direction of the initial image and the earth's latitude using satellite positioning or triangulation positioning; Based on the angle, the center coordinates of the target object, the first parameter of the first device, and the initial geographic coordinates, the actual width and height of the center of the target object from the center of the viewing angle of the first device are calculated: Based on the actual width and height, obtaining a distance between the acquisition position and the plane where the target object is located; Based on the distance between the acquisition position and the plane where the target object is located, the corresponding geographic coordinates of the target object are obtained.

8. The mobile terminal-oriented target positioning method according to claim 3, wherein fine-tuning parameters of the pre-trained first model and the pre-trained second model respectively comprises: All parameters of the first model are fine-tuned, and some parameters of the second model are fine-tuned.

9. The mobile terminal-oriented target positioning method according to claim 3, wherein the performing model conversion on the pre-trained first model and the pre-trained second model respectively further comprises: Normalize parameters of preset images.

Images