Target positioning methods, devices, equipment and media
By extracting and matching features from camera images in construction site and port scenarios, and correcting image offsets to calculate actual spatial coordinates, the problem of inaccurate target positioning caused by camera shake is solved, and the accuracy of target positioning is improved.
Patent Information
- Application Number
- CN202311303921.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-09
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-10-09
AI Technical Summary
In construction sites and ports, cameras are often installed at high positions, making them susceptible to wind and vehicle vibrations, which can lead to lower accuracy in target positioning.
By extracting and matching features from the image to be processed, the image offset coordinates are determined, and the actual spatial coordinates of the target are calculated in combination with the preset homography matrix, thereby correcting the image coordinates of the target.
It improves the accuracy of target positioning and enhances the precision of target location determination under wind and vehicle vibration environments.
Smart Images

Figure CN117333686B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of target positioning, and more particularly to a target positioning method, apparatus, device, and medium. Background Technology
[0002] With the development of technology, target localization methods that determine the position of a target in the world coordinate system based on its position in an image have been widely used in various fields.
[0003] In existing technologies, target localization typically requires taking pictures with a camera to obtain images. Then, based on the correspondence between the image coordinate system and the actual spatial coordinate system, the coordinates of the target in the image in the actual space are determined, thus achieving target localization.
[0004] However, in work scenarios such as construction sites and ports, cameras are generally installed at a high position, making them susceptible to shaking caused by wind and vehicle movement. This can lead to changes in the shooting range and lower accuracy in target positioning. Summary of the Invention
[0005] This application provides a target positioning method, apparatus, device, and medium to address the problem that existing target positioning methods, which directly use images captured by a camera for target positioning, result in low accuracy in target positioning.
[0006] In a first aspect, embodiments of this application provide a target localization method, including:
[0007] The acquired image to be processed is subjected to target detection processing to obtain at least one target and the image coordinates of each target;
[0008] The image to be processed is subjected to feature extraction processing to obtain at least one first feature point corresponding to the image to be processed and a first descriptor corresponding to each first feature point;
[0009] Based on the image coordinates of the first feature point and the first descriptor, as well as the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, the first feature point and the second feature point are matched to obtain at least one first feature matching point pair;
[0010] For each first feature matching point pair, the image coordinates of the feature points in the first feature matching point pair are subtracted to obtain the offset coordinates corresponding to the first feature matching point pair.
[0011] Calculate the image offset coordinates based on the offset coordinates corresponding to the at least one first feature matching point pair;
[0012] The actual spatial coordinates of each target are calculated based on the image coordinates of each target, the image offset coordinates, and the preset homography matrix.
[0013] In one specific implementation, calculating the image offset coordinates based on the offset coordinates corresponding to the at least one first feature matching point pair includes:
[0014] A density-based clustering algorithm is used to cluster the offset coordinates corresponding to the at least one first feature matching point pair to obtain at least one coordinate class;
[0015] The coordinate class with the largest offset coordinate among the at least one coordinate class shall be the result coordinate class;
[0016] The average value of the offset coordinates in the resulting coordinate class is used as the image offset coordinate.
[0017] In one specific embodiment, after performing feature extraction processing on the image to be processed to obtain at least one first feature point corresponding to the image to be processed and a first descriptor corresponding to each first feature point, the method further includes:
[0018] Delete the first feature points that belong to a preset position range in the image to be processed from the at least one first feature points to obtain the third feature point;
[0019] Delete the second feature points that belong to the preset position range in the preset reference image from the second feature points to obtain the fourth feature point;
[0020] Accordingly, the step of matching the first feature point and the second feature point based on the image coordinates of the first feature point and the first descriptor, and the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, to obtain at least one first feature matching point pair, includes:
[0021] Based on the image coordinates of the third feature point and the first descriptor corresponding to each third feature point, and the image coordinates of the fourth feature point and the second descriptor corresponding to each fourth feature point, the third feature point and the fourth feature point are matched to obtain at least one first feature matching point pair.
[0022] In one specific embodiment, before performing target detection processing on the acquired image to be processed to obtain at least one target and the image coordinates of each target, the method further includes:
[0023] Obtain the original image set;
[0024] Feature extraction processing is performed on each image to be selected in the original image set to obtain at least one second feature point and a second descriptor corresponding to each second feature point;
[0025] For every two images to be selected in the original image set, the second feature points corresponding to the two images to be selected are matched according to the image coordinates of the second feature points corresponding to the two images to be selected and the second descriptor corresponding to the second feature points, so as to obtain the second feature matching point pair corresponding to the two images to be selected;
[0026] Based on the image coordinates of the fifth feature point in the second feature matching point pair, calculate the average coordinate distance between the matching point pairs corresponding to the two images to be selected;
[0027] If there is no average coordinate distance of matching point pairs greater than a preset distance threshold among all matching point pairs, then for each image to be selected, the jitter parameter value corresponding to the image to be selected is calculated based on the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected.
[0028] The image with the smallest jitter parameter value from the original image set is selected as the preset reference image.
[0029] In one specific implementation, the step of calculating the jitter parameter value corresponding to the image to be selected based on the second feature matching point pairs corresponding to the image to be selected and each image in the original image set excluding the image to be selected includes:
[0030] Calculate the Euclidean distance between the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected;
[0031] The sum of the squares of the Euclidean distances is used as the jitter parameter value corresponding to the image to be selected.
[0032] In one specific embodiment, the method further includes:
[0033] Acquire a set of images of the moving target;
[0034] Moving target detection is performed on the moving target image set to obtain initial target points;
[0035] The initial target points belonging to the preset target category are used as the result target points;
[0036] Cluster the target points of the results to determine the baseline points of the results;
[0037] Obtain the actual spatial coordinates of the reference point;
[0038] The preset homography matrix is generated based on the image coordinates of the result reference point and the actual spatial coordinates of the result reference point.
[0039] In one specific implementation, the step of clustering the target points to determine the baseline points includes:
[0040] Based on the preset number of clusters and the preset clustering distance, the target points are clustered to obtain at least four target point classes;
[0041] Use the center point of each target point class as the initial reference point;
[0042] Generate the bounding rectangle of the initial reference point;
[0043] The union of the vertices of the circumscribed rectangle and the initial reference point is taken as the resulting reference point.
[0044] Secondly, embodiments of this application provide a target positioning device, comprising:
[0045] The target detection module is used to perform target detection processing on the acquired image to be processed, so as to obtain at least one target and the image coordinates of each target;
[0046] The feature extraction module is used to perform feature extraction processing on the image to be processed to obtain at least one first feature point corresponding to the image to be processed and a first descriptor corresponding to each first feature point;
[0047] The matching module is used to match the first feature point and the second feature point according to the image coordinates of the first feature point and the first descriptor, as well as the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, to obtain at least one first feature matching point pair;
[0048] Processing module, used for:
[0049] For each first feature matching point pair, the image coordinates of the feature points in the first feature matching point pair are subtracted to obtain the offset coordinates corresponding to the first feature matching point pair.
[0050] Calculate the image offset coordinates based on the offset coordinates corresponding to the at least one first feature matching point pair;
[0051] The actual spatial coordinates of each target are calculated based on the image coordinates of each target, the image offset coordinates, and the preset homography matrix.
[0052] Thirdly, embodiments of this application provide an electronic device, including:
[0053] Processor, memory, communication interface;
[0054] The memory is used to store the executable instructions of the processor;
[0055] The processor is configured to execute the target localization method according to any one of the first aspects by executing the executable instructions.
[0056] Fourthly, embodiments of this application provide a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the target localization method described in any of the first aspects.
[0057] The target localization method, apparatus, device, and medium provided in this application extract features from the image to be processed, match the extracted first feature points with the corresponding second feature points of a preset reference image, and then calculate the image offset coordinates based on the matched feature points. After target detection processing of the image to be processed, the actual spatial coordinates of the target can be obtained based on the target image coordinates obtained from target detection, combined with the image offset coordinates and a preset homography matrix. This solution effectively improves the accuracy of target localization by determining the image offset coordinates and calculating the actual spatial coordinates of the target. Attached Figure Description
[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0059] Figure 1 A flowchart illustrating an embodiment of the target positioning method provided in this application;
[0060] Figure 2 A flowchart illustrating Embodiment 2 of the target positioning method provided in this application;
[0061] Figure 3 A flowchart illustrating Embodiment 3 of the target positioning method provided in this application;
[0062] Figure 4 A schematic diagram of the structure of an embodiment of the target positioning device provided in this application;
[0063] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation
[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments made by those skilled in the art under the guidance of these embodiments are within the scope of protection of this application.
[0065] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0066] With the development of technology, target positioning has been widely used in scenarios such as transportation, construction sites, and ports, which can improve operational efficiency.
[0067] In existing technologies, camera calibration can be performed to determine camera parameters such as intrinsic parameters, distortion, and extrinsic parameters, thereby determining the correspondence between the image coordinate system and the actual spatial coordinate system.
[0068] Alternatively, a neural network model can be used to learn the correspondence between the image coordinate system and the actual spatial coordinate system from a large amount of image data; or the correspondence between the image coordinate system and the actual spatial coordinate system can be determined by calculating the homography matrix.
[0069] Then, after taking a picture with a camera to obtain an image, the image coordinates of the target in the image are determined. Based on this correspondence, the coordinates of the target in the image in the actual space are determined, thus achieving target localization.
[0070] However, in work scenarios such as construction sites and ports, cameras are generally installed at a high position, making them susceptible to shaking caused by wind and vehicle movement, which in turn causes changes in the shooting range. If the image coordinates of the target are still determined through this image for target positioning, the accuracy of target positioning will be low.
[0071] To address the problems existing in the prior art, the inventors, during their research on target localization methods, discovered that to improve the accuracy of target localization, after acquiring the image to be processed, feature points are extracted and then matched with feature points of a preset reference image. Based on the image coordinates of the successfully matched feature points, the image offset coordinates are determined. Then, based on the pixel coordinates of the target in the image to be processed, combined with the image offset coordinates, the image coordinates can be corrected. Finally, using a preset homography matrix, the actual spatial coordinates of the target can be obtained, effectively improving the accuracy of target localization. Based on the above inventive concept, the target localization scheme of this application was designed.
[0072] The target localization method in this application can be executed by a computer, or by a server, camera, or other devices. This application does not limit these devices. The following explanation uses a computer as an example.
[0073] The following provides examples illustrating the application scenarios of the target localization method provided in this application.
[0074] For example, in this application scenario, in a port operation scenario, the camera is installed at a high position on a high-mast light or steel frame. The camera transmits the captured images to a computer, which performs target positioning to determine the location of pedestrians and vehicles for scheduling.
[0075] After the computer acquires the image to be processed, it can perform target detection processing to obtain at least one target and the image coordinates of each target.
[0076] Because the wind is generally strong in port operation scenarios, the camera is easily affected by the wind and shakes. Also, the vehicles in the port are generally heavy, and when the vehicles pass by high-mast lights or steel frames, it will also cause the camera to shake. Therefore, the computer needs to correct the image coordinates of the targets in the image to be processed.
[0077] After acquiring the image to be processed, the computer needs to perform feature extraction processing, match the extracted feature points with the feature points in the preset reference image, and then determine the image offset coordinates based on the image coordinates of the first successfully matched feature point.
[0078] The computer then subtracts the image coordinates of each target from the image offset coordinates to obtain the corrected coordinates. Combined with the preset homography matrix, the actual spatial coordinates of each target can be obtained.
[0079] The computer can then display the location of each target on a real-world spatial map based on its actual spatial coordinates, so that staff can view or schedule it.
[0080] It should be noted that the above scenario is only an example of an application scenario provided by the embodiments of this application. The embodiments of this application do not limit the actual form of the various devices included in the scenario, nor do they limit the interaction method between devices. In the specific application of the solution, it can be set according to actual needs.
[0081] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0082] Figure 1 This is a flowchart illustrating an embodiment of the target localization method provided in this application. This embodiment describes how a computer extracts feature points from an image to be processed, matches them with feature points in a preset reference image, determines image offset coordinates based on the matching results, and then determines the actual spatial coordinates of the target based on the image offset coordinates. The method in this embodiment can be implemented using software, hardware, or a combination of both. Figure 1 As shown, the target localization method specifically includes the following steps:
[0083] S101: Perform target detection processing on the acquired image to be processed to obtain at least one target and the image coordinates of each target.
[0084] In this step, after the camera captures the image to be processed, it is transmitted to the computer for processing. The computer performs target detection processing on the acquired image to obtain at least one target and the image coordinates of each target.
[0085] It should be noted that the object detection processing can be performed using the R-CNN (Region with CNN Feature) algorithm, the YOLO (You Only Look Once) algorithm, or the DETR (Detection Transformer) algorithm. This application does not limit the method of object detection processing; it can be set according to the actual situation.
[0086] It should be noted that after obtaining the target by performing target detection processing on the image to be processed, targets that do not belong to the preset target category can be deleted, and the remaining targets can be used for target localization in the future. The preset target category can be vehicles, pedestrians, animals, etc. This application embodiment does not limit the preset target category and can be set according to the actual situation.
[0087] S102: Perform feature extraction processing on the image to be processed to obtain at least one first feature point corresponding to the image to be processed and a first descriptor corresponding to each first feature point.
[0088] In this step, after the computer obtains the image to be processed, a feature extraction algorithm is used to perform feature extraction processing on the image to correct the targets in the image, obtaining at least one first feature point corresponding to the image and a first descriptor corresponding to each first feature point. The descriptor is a vector that represents the features of the feature point.
[0089] It should be noted that the feature extraction algorithm can be the Scale Invariant Feature Transform (SIFT) algorithm, or the Speeded-Up Robust Features (SURF) algorithm, the ORB (Oriented Fast and Rotated BRIEF) algorithm, etc. The embodiments of this application do not limit the feature extraction algorithm, and it can be determined according to the actual situation.
[0090] S103: Match the first feature point and the second feature point according to the image coordinates and the first descriptor of the first feature point, as well as the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, to obtain at least one first feature matching point pair.
[0091] In this step, after the computer obtains the first feature point and the first descriptor, it can use a feature matching algorithm to match the first feature point and the second feature point based on the image coordinates of the first feature point and the first descriptor, as well as the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, to obtain at least one first feature matching point pair.
[0092] It should be noted that the feature matching algorithm can be a brute-froce matcher algorithm, or an algorithm from the Fast Library for Approximate Nearest Neighbors (FLANN), a cross-matching algorithm, etc. The embodiments of this application do not limit the feature matching algorithm, and it can be determined according to the actual situation.
[0093] Optionally, at least one first feature point that belongs to a preset position range in the image to be processed is deleted to obtain a third feature point; and a second feature point that belongs to a preset position range in a preset reference image is deleted to obtain a fourth feature point.
[0094] Then, based on the image coordinates of the third feature point and the first descriptor corresponding to each third feature point, and the image coordinates of the fourth feature point and the second descriptor corresponding to each fourth feature point, the third feature point and the fourth feature point are matched to obtain at least one first feature matching point pair, which can improve the accuracy of matching.
[0095] It should be noted that if the distance of a feature point from the left or right side of the image is less than a preset first distance, or the distance of a feature point from the top or bottom of the image is less than a preset second distance, then the feature point belongs to a preset position range. The preset first distance and the preset second distance can be 5 pixels, 6 pixels, 10 pixels, etc. This application embodiment does not limit the preset first distance and the preset second distance, and they can be determined according to the actual situation.
[0096] S104: For each first feature matching point pair, subtract the image coordinates of the feature points in the first feature matching point pair to obtain the offset coordinates corresponding to the first feature matching point pair.
[0097] In this step, after the computer obtains the first feature matching point pair, for each first feature matching point pair, the image coordinates of the feature points in the first feature matching point pair are subtracted to obtain the offset coordinates corresponding to the first feature matching point pair.
[0098] For example, the image coordinates of the first feature point in the first feature matching point pair are: The image coordinates of the second feature point are: Then the offset coordinates corresponding to the first feature matching point pair are (Δx) i ,Δy i ),in,
[0099] It should be noted that when the feature matching algorithm outputs the first feature matching point pair, it can also determine the matching degree. The first feature matching point pair can be sorted according to the matching degree. If the number of offset coordinates obtained is greater than the preset number of rejections, the first feature matching point pair with the first preset number of offsets is selected to calculate the offset coordinates.
[0100] The preset rejection quantity can be 500, 1000, 5000, etc., and the preset offset quantity can be 100, 200, 300, etc. This application embodiment does not limit the preset rejection quantity and the preset offset quantity, and can be set according to the actual situation.
[0101] S105: Calculate the image offset coordinates based on the offset coordinates corresponding to at least one first feature matching point pair.
[0102] In this step, after the computer obtains the offset coordinates corresponding to the first feature matching point pair, it can calculate the image offset coordinates based on the offset coordinates corresponding to at least one first feature matching point pair.
[0103] Specifically, a density-based clustering algorithm is used to cluster the offset coordinates corresponding to at least one first feature matching point pair to obtain at least one coordinate class.
[0104] The coordinate class with the largest offset coordinates among at least one coordinate class is taken as the result coordinate class; then the average value of the offset coordinates in the result coordinate class is taken as the image offset coordinate.
[0105] For example, the coordinates in the resulting coordinate class are, in order: (Δx1, Δy1), ..., (Δx m ,Δy m Then the image offset coordinates are: in, m is the number of coordinates in the result coordinate class.
[0106] It should be noted that the density-based clustering algorithm can be DBSCAN (Density-Based Spatial Clustering of Applications with Noise), OPTICS (Ordering points to identify the clustering structure), DENCLUE (Density based Clustering), etc. This application does not limit the density-based clustering algorithm, and it can be determined according to the actual situation.
[0107] It should be noted that when using the DBSCAN algorithm for clustering, the parameter eps can be set to a smaller preset value to make the clustering results more consistent with the shaking scenario. The preset parameter value can be 2 pixels, 3 pixels, 5 pixels, etc. This application embodiment does not limit the preset parameter value and can be determined according to the actual situation.
[0108] It should be noted that the execution order of steps S101 and S102-S105 can be: step S101 is executed first, followed by steps S102-S105; or steps S102-S105 are executed first, followed by step S101; or steps S101 and S102-S105 are executed simultaneously. This embodiment does not limit the execution order of steps S101 and S102-S105, and can be set according to actual circumstances.
[0109] S106: Calculate the actual spatial coordinates of each target based on its image coordinates, image offset coordinates, and preset homography matrix.
[0110] In this step, after the computer determines the image coordinates and image offset coordinates of the target in the image to be processed, for each target, the image coordinates of the target are subtracted from the image offset coordinates to obtain the corrected coordinates of the target. Combined with the preset homography matrix, the actual spatial coordinates of the target can be calculated.
[0111] For example, the target's image coordinates are (x, y), and the image offset coordinates are... Therefore, the corrected coordinates of the target are (x′, y′), where, The homography matrix is a 3x3 matrix H. The corrected coordinates are padded to (x... ′ ,y ′ From ,1), we can determine that (X,Y,1)=H*(x ′ ,y ′ ,1), the actual spatial coordinates of the target are (X,Y).
[0112] The target localization method provided in this embodiment extracts features from the image to be processed, matches the extracted first feature points with the corresponding second feature points of a preset reference image, and then calculates the image offset coordinates based on the matched feature points. After target detection processing of the image to be processed, the actual spatial coordinates of the target can be obtained based on the target image coordinates obtained from target detection, combined with the image offset coordinates and a preset homography matrix. Compared with the prior art that directly uses the target's image coordinates to calculate the target's actual spatial coordinates, this solution corrects the target's image coordinates by determining the image offset coordinates, and then uses the corrected coordinates to calculate the target's actual spatial coordinates, effectively improving the accuracy of target localization.
[0113] Figure 2 This is a flowchart illustrating a second embodiment of the target localization method provided in this application. Based on the above embodiments, this application describes the acquisition of a preset reference image before the computer performs target localization. Figure 2 As shown, the target localization method specifically includes the following steps:
[0114] S201: Obtain the original image set.
[0115] In this step, before the computer can locate the target, it needs to determine a preset reference image, which requires acquiring the original image set first.
[0116] The camera sends the recorded video to a computer, where staff can select video segments with less shaking and fewer moving targets. The computer then extracts images from these video segments to form the original image set.
[0117] After the camera sends the recorded video to the computer, the computer can directly extract images from the video to form an original image set.
[0118] S202: Perform feature extraction processing on each image to be selected in the original image set to obtain at least one second feature point and a second descriptor corresponding to each second feature point.
[0119] In this step, after the computer obtains the original image set, it needs to determine whether the jitter level of the original image set meets the requirements. It needs to perform feature extraction processing on each image to be selected in the original image set to obtain at least one second feature point and a second descriptor corresponding to each second feature point.
[0120] It should be noted that this step is similar to step S102 in Embodiment 1, and will not be described again here.
[0121] S203: For every two images to be selected in the original image set, the second feature points corresponding to the two images to be selected are matched according to the image coordinates of the second feature points and the second descriptors corresponding to the second feature points, to obtain the second feature matching point pairs corresponding to the two images to be selected.
[0122] It should be noted that this step is similar to step S103 in Embodiment 1, and will not be described again here.
[0123] S204: Based on the image coordinates of the fifth feature point in the second feature matching point pair, calculate the average coordinate distance between the matching point pairs corresponding to the two images to be selected.
[0124] In this step, after the computer determines the second feature matching point pair, it calculates the average coordinate distance between the matching point pairs corresponding to the two images to be selected based on the image coordinates of the fifth feature point in the second feature matching point pair.
[0125] For example, there are m pairs of second feature matching points corresponding to the two images to be selected, where the image coordinates of the two fifth feature points in the second feature matching point pairs are respectively: (x ik ,y ik ) and (x jk ,y jk The average coordinate distance between the matching point pairs corresponding to the two images to be selected is... in,
[0126] It should be noted that when the feature matching algorithm outputs the second feature matching point pair, it can also determine the matching degree. The second feature matching point pair can be sorted according to the matching degree. If the number of second feature matching point pairs obtained is greater than the preset number of eliminations, the average coordinate distance of the matching point pair is calculated by selecting the second feature matching point pairs with the first preset offset.
[0127] The preset rejection quantity can be 500, 1000, 5000, etc., and the preset offset quantity can be 100, 200, 300, etc. This application embodiment does not limit the preset rejection quantity and the preset offset quantity, and can be set according to the actual situation.
[0128] S205: If there is no average coordinate distance greater than the preset distance threshold among all average coordinate distances, then for each image to be selected, calculate the jitter parameter value corresponding to the image to be selected based on the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected.
[0129] In this step, after the computer determines the average coordinate distance between matching point pairs corresponding to each pair of images to be selected, it determines whether there is an average coordinate distance greater than a preset distance threshold among all the average coordinate distances, so as to determine whether the jitter level of the original image set meets the requirements.
[0130] If none of the average coordinate distances is greater than the preset distance threshold, it means that the jitter level of the original image set meets the requirements. Then, for each image to be selected, the jitter parameter value corresponding to the image to be selected is calculated based on the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected.
[0131] It should be noted that the preset distance threshold can be 2 pixels, 3 pixels, 5 pixels, etc. This application embodiment does not limit the preset distance threshold, and it can be set according to the actual situation.
[0132] Specifically, first calculate the Euclidean distance between the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected.
[0133] The Euclidean distance of the second feature matching point pair refers to the Euclidean distance between the image coordinates of the two second feature points in the second feature matching point pair.
[0134] The sum of the squares of all the obtained Euclidean distances is then used as the jitter parameter value corresponding to the image to be selected.
[0135] It should be noted that when the feature matching algorithm outputs the second feature matching point pair, it can also determine the matching degree. The second feature matching point pair can be sorted according to the matching degree. If the number of second feature matching point pairs obtained is greater than the preset number of rejections, then the second feature matching point pairs with the first preset offset number are selected to calculate the jitter parameter value.
[0136] It should be noted that if there is an average coordinate distance greater than the preset distance threshold among all the average coordinate distances, it means that the jitter level of the original image set does not meet the requirements. In this case, it is necessary to reacquire the original image set and re-determine whether there is an average coordinate distance greater than the preset distance threshold among all the average coordinate distances until there is no average coordinate distance greater than the preset distance threshold among all the average coordinate distances, and then calculate the jitter parameter value.
[0137] S206: Select the image with the smallest jitter parameter value from the original image set as the preset reference image.
[0138] In this step, after the computer obtains the jitter parameter value corresponding to each image to be selected, it selects the image with the smallest jitter parameter value from the original image set as the preset reference image.
[0139] It should be noted that the computer also needs to store the second feature points corresponding to the preset reference image and the second descriptors corresponding to the second feature points. Alternatively, the preset reference image can be re-processed for feature extraction, and the extracted second feature points and the second descriptors corresponding to the second feature points can be stored.
[0140] It should be noted that when the camera's installation location or orientation changes, the preset reference image needs to be re-determined following the steps described above.
[0141] The target localization method provided in this embodiment performs feature extraction processing on the acquired original image set, and then performs feature point matching. It calculates the average coordinate distance of the matching point pairs. When there is no average coordinate distance of the matching point pairs greater than a preset distance threshold among all the average coordinate distances of the matching point pairs, it calculates the jitter parameter value and selects the candidate image with the smallest jitter parameter value as the preset reference image, so that the jitter of the selected preset reference image is small.
[0142] Figure 3 This is a flowchart illustrating a third embodiment of the target positioning method provided in this application. Based on the above embodiments, this application describes the generation of a preset homography matrix after the computer determines a preset reference image. For example... Figure 3 As shown, the target localization method specifically includes the following steps:
[0143] S301: Acquire the image set of the moving target.
[0144] In this step, before the computer can locate the target, it needs to determine the preset homography matrix, which requires acquiring a set of moving target images first.
[0145] The camera sends the recorded video to the computer, where staff can select video segments with a high number of moving targets. The computer then extracts images from these segments to create a set of moving target images.
[0146] S302: Perform moving target detection on the moving target image set to obtain initial target points.
[0147] In this step, after the computer obtains the set of moving target images, it performs moving target detection on the images to obtain the initial target points.
[0148] It should be noted that this step is similar to step S101 in Embodiment 1, and will not be described again here.
[0149] S303: Use the initial target point that belongs to the preset target category as the result target point.
[0150] In this step, after the computer obtains the initial target points, it also needs to use the initial target points belonging to the preset target categories as the result target points. Subsequent target localization using the preset homography matrix determined based on the result target points can improve the accuracy of target localization.
[0151] It should be noted that the preset target category can be vehicles, pedestrians, animals, etc. This application embodiment does not limit the preset target category and can be set according to the actual situation.
[0152] S304: Perform clustering on the target points to determine the baseline points.
[0153] In this step, after the computer obtains the target points, it performs clustering on the target points to determine the baseline points.
[0154] Specifically, based on the preset number of clusters and the preset clustering distance, the target points are clustered to obtain at least four target point classes, and the distance between the center points of any two target point classes is greater than the preset clustering distance.
[0155] It should be noted that the preset number of clusters is a number greater than or equal to 4, such as 4, 5, or 9. The preset cluster distance is 1 / K of the shorter side of the images in the moving target image set, where K is the preset number of clusters. This application does not limit the preset number of clusters or the preset cluster distance; they can be set according to actual conditions.
[0156] It should be noted that the clustering method can be K-means, K-medians, K-sums, etc. This application does not limit the clustering method and can be set according to the actual situation.
[0157] The center point of each target point class is used as the initial reference point; then the bounding rectangle of the initial reference point is generated, and the union of the vertices of the bounding rectangle and the initial reference point is used as the result reference point.
[0158] It should be noted that if the side length of the circumscribed rectangle is less than a preset side length threshold, the length of that side will be increased to the preset side length threshold. The preset side length threshold can be 100 pixels, 200 pixels, 500 pixels, etc. This application embodiment does not limit the preset side length threshold and can be set according to the actual situation.
[0159] It should be noted that the initial reference point can also be used as the result reference point.
[0160] S305: Obtain the actual spatial coordinates of the result reference point.
[0161] In this step, after the computer obtains the result reference point, it can mark the result reference point on the preset reference image for display. After the staff checks it, they can inspect the result reference point. If the result reference point is not on the same plane in the actual space, it can be adjusted to make it on the same plane in the actual space.
[0162] After the staff determines the actual spatial coordinates of each result reference point, they input them into the computer, which can then obtain the actual spatial coordinates of the result reference point.
[0163] S306: Generate a preset homography matrix based on the image coordinates of the result reference point and the actual spatial coordinates of the result reference point.
[0164] In this step, after the computer determines the image coordinates and actual spatial coordinates of the result reference points, since the number of result reference points is greater than 4, a preset homography matrix can be generated based on the image coordinates and actual spatial coordinates of the result reference points.
[0165] It should be noted that when the camera's installation location or orientation changes, the preset homography matrix needs to be redefined following the steps described above.
[0166] The target localization method provided in this embodiment determines the target points in the moving target image set through moving target detection, then performs clustering processing to obtain the target reference points. Based on the image coordinates and actual spatial coordinates of the target reference points, a preset homography matrix is generated, which improves the accuracy of subsequent target localization using the preset homography matrix. Furthermore, camera calibration is unnecessary. For the homography matrix, which reflects the correspondence between the image coordinate system and the actual spatial coordinate system, errors are not caused by the distance between the camera and the target, resulting in higher accuracy and lower cost in generating the preset homography matrix. Model training is also unnecessary, improving the efficiency and cost of generating the preset homography matrix.
[0167] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.
[0168] Figure 4 This is a schematic diagram of the structure of an embodiment of the target positioning device provided in this application. Figure 4 As shown, the target positioning device 40 includes:
[0169] The target detection module 41 is used to perform target detection processing on the acquired image to be processed, and obtain at least one target and the image coordinates of each target;
[0170] Feature extraction module 42 is used to perform feature extraction processing on the image to be processed to obtain at least one first feature point corresponding to the image to be processed and a first descriptor corresponding to each first feature point;
[0171] Matching module 43 is used to match the first feature point and the second feature point according to the image coordinates of the first feature point and the first descriptor, as well as the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, to obtain at least one first feature matching point pair;
[0172] Processing module 44 is used for:
[0173] For each first feature matching point pair, the image coordinates of the feature points in the first feature matching point pair are subtracted to obtain the offset coordinates corresponding to the first feature matching point pair.
[0174] Calculate the image offset coordinates based on the offset coordinates corresponding to the at least one first feature matching point pair;
[0175] The actual spatial coordinates of each target are calculated based on the image coordinates of each target, the image offset coordinates, and the preset homography matrix.
[0176] Furthermore, the processing module 44 is specifically used for:
[0177] A density-based clustering algorithm is used to cluster the offset coordinates corresponding to the at least one first feature matching point pair to obtain at least one coordinate class;
[0178] The coordinate class with the largest offset coordinate among the at least one coordinate class shall be the result coordinate class;
[0179] The average value of the offset coordinates in the resulting coordinate class is used as the image offset coordinate.
[0180] Furthermore, the processing module 44 is also used for:
[0181] Delete the first feature points that belong to a preset position range in the image to be processed from the at least one first feature points to obtain the third feature point;
[0182] Delete the second feature points that belong to the preset position range in the preset reference image from the second feature points to obtain the fourth feature point;
[0183] Furthermore, the matching module 43 is specifically used for:
[0184] Based on the image coordinates of the third feature point and the first descriptor corresponding to each third feature point, and the image coordinates of the fourth feature point and the second descriptor corresponding to each fourth feature point, the third feature point and the fourth feature point are matched to obtain at least one first feature matching point pair.
[0185] Furthermore, the processing module 44 is also used to acquire the original image set;
[0186] Furthermore, the feature extraction module 42 is also used to perform feature extraction processing on each image to be selected in the original image set to obtain at least one second feature point corresponding to each image to be selected and a second descriptor corresponding to each second feature point;
[0187] Furthermore, the matching module 43 is also used to match the second feature points corresponding to the two images to be selected for every two images in the original image set, based on the image coordinates of the second feature points corresponding to the two images to be selected and the second descriptor corresponding to the second feature points, to obtain the second feature matching point pair corresponding to the two images to be selected;
[0188] Furthermore, the processing module 44 is also used for:
[0189] Based on the image coordinates of the fifth feature point in the second feature matching point pair, calculate the average coordinate distance between the matching point pairs corresponding to the two images to be selected;
[0190] If there is no average coordinate distance of matching point pairs greater than a preset distance threshold among all matching point pairs, then for each image to be selected, the jitter parameter value corresponding to the image to be selected is calculated based on the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected.
[0191] The image with the smallest jitter parameter value from the original image set is selected as the preset reference image.
[0192] Furthermore, the processing module 44 is also used for:
[0193] Calculate the Euclidean distance between the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected;
[0194] The sum of the squares of the Euclidean distances is used as the jitter parameter value corresponding to the image to be selected.
[0195] Furthermore, the processing module 44 is also used to acquire a set of images of the moving target;
[0196] Furthermore, the target detection module 41 is also used to perform moving target detection on the moving target image set to obtain initial target points;
[0197] Furthermore, the processing module 44 is also used for:
[0198] The initial target points belonging to the preset target category are used as the result target points;
[0199] Cluster the target points of the results to determine the baseline points of the results;
[0200] Obtain the actual spatial coordinates of the reference point;
[0201] The preset homography matrix is generated based on the image coordinates of the result reference point and the actual spatial coordinates of the result reference point.
[0202] Furthermore, the processing module 44 is also used for:
[0203] Based on the preset number of clusters and the preset clustering distance, the target points are clustered to obtain at least four target point classes;
[0204] Use the center point of each target point class as the initial reference point;
[0205] Generate the bounding rectangle of the initial reference point;
[0206] The union of the vertices of the circumscribed rectangle and the initial reference point is taken as the resulting reference point.
[0207] The target positioning device provided in this embodiment is used to execute the technical solution in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.
[0208] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 5 As shown, the electronic device 50 includes:
[0209] Processor 51, memory 52, and communication interface 53;
[0210] The memory 52 is used to store the executable instructions of the processor 51;
[0211] The processor 51 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the executable instructions.
[0212] Optionally, the memory 52 can be either standalone or integrated with the processor 51.
[0213] Optionally, when the memory 52 is a device independent of the processor 51, the electronic device 50 may further include:
[0214] Bus 54, memory 52 and communication interface 53 are connected to processor 51 through bus 54 and complete communication with each other. Communication interface 53 is used to communicate with other devices.
[0215] Optionally, the communication interface 53 can be implemented using a transceiver. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write databases, and read-only databases). The memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.
[0216] Bus 54 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.
[0217] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0218] The electronic device is used to execute the technical solutions in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.
[0219] This application also provides a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the technical solutions provided in any of the foregoing method embodiments.
[0220] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the technical solutions provided in any of the foregoing method embodiments.
[0221] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0222] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A target positioning method characterized by, include: The acquired image to be processed is subjected to target detection processing to obtain at least one target and the image coordinates of each target; The image to be processed is subjected to feature extraction processing to obtain at least one first feature point corresponding to the image to be processed and a first descriptor corresponding to each first feature point; Based on the image coordinates of the first feature point and the first descriptor, as well as the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, the first feature point and the second feature point are matched to obtain at least one first feature matching point pair; For each first feature matching point pair, the image coordinates of the feature points in the first feature matching point pair are subtracted to obtain the offset coordinates corresponding to the first feature matching point pair. Calculate the image offset coordinates based on the offset coordinates corresponding to the at least one first feature matching point pair; The actual spatial coordinates of each target are calculated based on the image coordinates of each target, the image offset coordinates, and the preset homography matrix. The step of calculating the image offset coordinates based on the offset coordinates corresponding to the at least one first feature matching point pair includes: A density-based clustering algorithm is used to cluster the offset coordinates corresponding to the at least one first feature matching point pair to obtain at least one coordinate class; The coordinate class with the largest offset coordinate among the at least one coordinate class shall be the result coordinate class; The average value of the offset coordinates in the resulting coordinate class is used as the image offset coordinate.
2. The method of claim 1, wherein, After performing feature extraction processing on the image to be processed to obtain at least one first feature point corresponding to the image to be processed and a first descriptor corresponding to each first feature point, the method further includes: Delete the first feature points that belong to a preset position range in the image to be processed from the at least one first feature points to obtain the third feature point; Delete the second feature points that belong to the preset position range in the preset reference image from the second feature points to obtain the fourth feature point; Accordingly, the step of matching the first feature point and the second feature point based on the image coordinates of the first feature point and the first descriptor, and the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, to obtain at least one first feature matching point pair, includes: Based on the image coordinates of the third feature point and the first descriptor corresponding to each third feature point, and the image coordinates of the fourth feature point and the second descriptor corresponding to each fourth feature point, the third feature point and the fourth feature point are matched to obtain at least one first feature matching point pair.
3. The method of claim 1, wherein, Before performing target detection processing on the acquired image to be processed to obtain at least one target and the image coordinates of each target, the method further includes: Obtain the original image set; Feature extraction processing is performed on each image to be selected in the original image set to obtain at least one second feature point and a second descriptor corresponding to each second feature point; For every two images to be selected in the original image set, the second feature points corresponding to the two images to be selected are matched according to the image coordinates of the second feature points corresponding to the two images to be selected and the second descriptor corresponding to the second feature points, so as to obtain the second feature matching point pair corresponding to the two images to be selected; Based on the image coordinates of the fifth feature point in the second feature matching point pair, calculate the average coordinate distance between the matching point pairs corresponding to the two images to be selected; If there is no average coordinate distance of matching point pairs greater than a preset distance threshold among all matching point pairs, then for each image to be selected, the jitter parameter value corresponding to the image to be selected is calculated based on the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected. The image with the smallest jitter parameter value from the original image set is selected as the preset reference image.
4. The method of claim 3, wherein, The step of calculating the jitter parameter value corresponding to the image to be selected based on the second feature matching point pairs corresponding to the image to be selected and each image in the original image set excluding the image to be selected includes: Calculate the Euclidean distance between the second feature matching point pairs corresponding to the image to be selected and each image in the original image set except for the image to be selected; The sum of the squares of the Euclidean distances is used as the jitter parameter value corresponding to the image to be selected.
5. The method of claim 3, wherein, The method further includes: Acquire a set of images of the moving target; Moving target detection is performed on the moving target image set to obtain initial target points; The initial target points belonging to the preset target category are used as the result target points; Cluster the target points of the results to determine the baseline points of the results; Obtain the actual spatial coordinates of the reference point; The preset homography matrix is generated based on the image coordinates of the result reference point and the actual spatial coordinates of the result reference point.
6. The method of claim 5, wherein, The step of clustering the target points to determine the baseline points includes: Based on the preset number of clusters and the preset clustering distance, the target points are clustered to obtain at least four target point classes; Use the center point of each target point class as the initial reference point; Generate the bounding rectangle of the initial reference point; The union of the vertices of the circumscribed rectangle and the initial reference point is taken as the resulting reference point.
7. A target positioning device, characterized by include: The target detection module is used to perform target detection processing on the acquired image to be processed, so as to obtain at least one target and the image coordinates of each target; The feature extraction module is used to perform feature extraction processing on the image to be processed to obtain at least one first feature point corresponding to the image to be processed and a first descriptor corresponding to each first feature point; The matching module is used to match the first feature point and the second feature point according to the image coordinates of the first feature point and the first descriptor, as well as the image coordinates of the second feature point corresponding to the preset reference image and the second descriptor corresponding to each second feature point, to obtain at least one first feature matching point pair; Processing module, used for: For each first feature matching point pair, the image coordinates of the feature points in the first feature matching point pair are subtracted to obtain the offset coordinates corresponding to the first feature matching point pair. Calculate the image offset coordinates based on the offset coordinates corresponding to the at least one first feature matching point pair; The actual spatial coordinates of each target are calculated based on the image coordinates of each target, the image offset coordinates, and the preset homography matrix. The processing module is specifically used for: A density-based clustering algorithm is used to cluster the offset coordinates corresponding to the at least one first feature matching point pair to obtain at least one coordinate class; The coordinate class with the largest offset coordinate among the at least one coordinate class shall be the result coordinate class; The average value of the offset coordinates in the resulting coordinate class is used as the image offset coordinate.
8. An electronic device, comprising: include: Processor, memory, communication interface; The memory is used to store the executable instructions of the processor; The processor is configured to execute the target localization method according to any one of claims 1 to 6 by executing the executable instructions.
9. A readable storage medium, having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the target localization method according to any one of claims 1 to 6.
Citation Information
Patent Citations
ORB (oriented brief) image feature registration method based on LK (Lucas-Kanade) optical flow constraint
CN104751465A
Deviation correction method suitable for instrument image positioning of inspection robot
CN115797619A
