Multi-mode-based real-time target positioning and re-indicating method

Abstract

The invention belongs to the technical field of long-range target detection and positioning, and relates to a multi-modal-based real-time target positioning re-indication method, which comprises the following steps of: performing target identification and positioning on an original image acquired by a multispectral zoom camera in real time, endowing an ID value to a target and recording a GPS position; taking an area image of a target shot in the adjustable focal length range of each segment of the camera as a positioning reference template of a corresponding focal length level, so as to generate a multi-layer positioning reference template; when a target re-indication request is received, acquiring a GPS position corresponding to a re-indication target and a multi-layer positioning reference template, adjusting a camera visual angle position to enable a GPS value corresponding to an image center pixel to be the GPS position of the re-indication target, and taking an image at the moment as a mark image; and if the marked image is successfully matched with the positioning reference templates of all focal length levels participating in matching, the target indication is successful. The method has the advantages of high response speed, high stability and the like in target re-indication.

Description

Technical Field

[0001] The invention belongs to the technical field of distant target detection and positioning, and relates to a multi-modal real-time target positioning and re-indication method. Background Art

[0002] At present, the zoom multispectral camera carried by drones has become one of the mainstream technologies for target positioning and recognition. In the field of long-range target detection and positioning, in the process of searching for an indefinite number of targets, the searched targets often need to be quickly located again in the subsequent search process, that is, to quickly re-indicate the targets. Target re-indication is the re-positioning of the same target during the target positioning search process. It is a common operation in multi-target recognition and positioning, but the re-indication of the current target often means re-searching the target again, which has problems such as slow target re-indication speed, easy loss, and confusing targets. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a multi-modal real-time target positioning and re-indication method, which can complete the target rapid re-indication function and improve the accuracy and robustness of target recognition.

[0004] In order to solve the above technical problems, the multi-modal real-time target positioning and re-indication method of the present invention is as follows: Step 1: Perform target recognition and positioning on the original images collected in real time by the multi-spectral zoom camera carried by the drone, assign an ID value to the recognized target and record its GPS location; Step 2: dynamically adjust the focal length of the multispectral zoom camera, and use the target area image captured within the adjustable focal length range of each segment as the positioning reference template of the corresponding focal length level, so as to generate a multi-layer positioning reference template containing focal length information; Step 3: When a target re-indication request is received, first obtain the GPS position and multi-layer positioning reference template corresponding to the re-indication target; adjust the multi-spectral zoom camera viewing angle position so that the GPS value corresponding to the central pixel of the image is the GPS position of the re-indication target and use the image at this time as the marked image; if the marked image successfully matches the positioning reference templates of all focal length levels involved in the matching, the target re-indication is successful.

[0005] In the step 2, the gimbal posture is first adjusted so that the target in the image is at or close to the center, and then the image of the target area is captured within the adjustable focal length range of each segment as a positioning reference template for the corresponding focal length level.

[0006] In the step 2, the image of the target area is centered on the target coverage frame and is proportionally enlarged by 1.2-1.5 times.

[0007] In the step one, multiple targets are identified, one of which is located first, an ID value is assigned to it, and its GPS position is recorded, and then a multi-layer positioning reference template corresponding to the target is generated through step two; similarly, other identified targets are located in turn, ID values ​​are assigned to them, their GPS positions are recorded, and corresponding multi-layer positioning reference templates are generated.

[0008] In the step three, when a target re-indication request is received, if a re-indication target ID value is received at the same time, the corresponding GPS information and multi-layer positioning reference template are obtained according to the re-indication target ID value.

[0009] In the step 3, when a target re-indication request is received, if a re-indication target ID value is not received, the targets corresponding to the ID values ​​are re-indicated in order from the latest to the earliest in time.

[0010] In step 3, the method for matching the marked image with the positioning reference template is as follows: 1. Adjust the viewing angle of the multi-spectral zoom camera so that the GPS value corresponding to the central pixel of the image is the GPS position of the target, and use the image at this time as the marked image; 2. According to the current focal length of the multi-spectral zoom camera, select and then indicate the positioning reference template of the focal length level corresponding to the target and match it with the marked image; if the match fails, adjust the focal length of the multi-spectral zoom camera to a smaller focal length level and return to step 1; if the match is successful, move the registration position in the marked image to the center of the image and go to step 3; 3. Adjust the focal length of the multispectral zoom camera to a larger focal length level, obtain a new marked image, if the marked image fails to match the positioning reference template of the corresponding focal length level, adjust the focal length of the multispectral zoom camera to the previous focal length level, and return to step 1; if the match is successful, adjust the focal length of the multispectral zoom camera to a larger focal length level, obtain a new marked image, and then match the marked image with the positioning reference template of the corresponding focal length level; if the marked image successfully matches the positioning reference templates of all focal length levels involved in the matching, the target is indicated successfully.

[0011] Furthermore, if the number of failed matches between the marked image and the positioning reference template exceeds a set maximum number of matches, the target again indicates failure.

[0012] The marked image and the positioning reference template are matched using a feature matching algorithm, a histogram matching algorithm, a template-based matching algorithm, a deep learning-based matching algorithm or a coefficient normalization algorithm.

[0013] Beneficial effects: The multimodal real-time target positioning and re-indication method of the present invention combines a variety of sensor data (RGB images, infrared images, inertial navigation, positioning modules, etc.), uses artificial intelligence, traditional target matching and other technologies to perform target identification and positioning, and improves the accuracy and robustness of target identification. After identifying a target, while continuing to search for other targets, the camera image can be quickly switched back to the target at any time to complete the target rapid re-indication function. At the same time, in order to more effectively respond to multi-target re-indication, each identified target is ID-tagged, and the multi-target re-indication function can be completed. The present invention has the advantages of fast response speed and strong stability in target re-indication, and can be widely used in fire monitoring, intelligent transportation, automatic driving and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is an overall flow chart of the multi-modal real-time target positioning and re-indication method of the present invention.

[0015] Figure 2 It is an image in which target A and target B are recognized.

[0016] Figure 3 , Figure 4 , Figure 5 It is the three-layer positioning reference template corresponding to target A.

[0017] Figure 6 This is an image in which target B and target C are recognized.

[0018] Figure 7 , Figure 8 , Fig. 9 It is the three-layer positioning reference template corresponding to target B.

[0019] Fig.10 is the image of target C.

[0020] Fig.11 , Fig.12 , Fig.13 It is the three-layer positioning reference template corresponding to target C. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention rather than to limit the present invention.

[0022] In the description of the embodiments, the terms "upper", "lower", "left", "right" and other directions or positional relationships are based on the directions or positional relationships shown in the drawings, and are only for the convenience of description and simplified operation, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used to distinguish in the description and have no special meaning.

[0023] Fast and accurate target positioning is the key to realizing intelligent perception. Traditional single positioning methods, such as GPS and visual odometer, are greatly affected by environmental factors and have low positioning accuracy. To solve this problem, this method proposes a target re-indication method based on a multi-layer positioning template, which can comprehensively utilize multi-dimensional templates to quickly re-position the target.

[0024] like Figure 1 As shown, the multi-modal real-time target positioning and re-indication method of the present invention specifically includes the following steps: Step 1: Use the multi-spectral zoom camera carried by the drone to collect real-time images. After filtering, correcting, aligning and other pre-processing of the collected original images, use general computer vision technology to detect and extract features, identify and locate targets, and determine the specific location and size of each target; Figure 2 As shown in FIG. 1 , the target detected in the image may be one or more; firstly, manually indicate or automatically annotate target A, assign it an ID value id1, and record its GPS position; the multi-spectral zoom camera sets multiple adjustable focal ranges according to the imaging effect; collect image data of the area where target A is located, adjust the gimbal posture during collection, so that target A in the image is at the center or close to the center, dynamically adjust the exposure time, aperture, focal length, etc. of the multi-spectral zoom camera, and segment the camera's adjustable focal range using the segment range set by the software as the image acquisition level, and use the image of the area where target A is located taken within each segmented adjustable focal range as the positioning reference template of the corresponding focal level, so as to generate a multi-layer positioning reference template containing focal length information, such as Figure 3 , Figure 4 , Figure 5 As shown, the image 1 of the area where the target A is located is centered on the target coverage frame 2 and is proportionally enlarged by 1.2-1.5 times; Step 2: Use the multispectral zoom camera to collect real-time images and locate the identified target B, such as Figure 6 As shown, assign it an ID value id2, record its GPS position, and use the same method as that of generating the multi-layer positioning reference template corresponding to target A to obtain the multi-layer positioning reference template corresponding to target B, as shown in Figure 7 , Figure 8 , Fig. 9As shown; Similarly, the identified target C is located, as shown Fig.10 As shown, assign an ID value id3 to it, record its GPS position, and generate a corresponding multi-layer positioning reference template containing focal length information, such as Fig.11 , Fig.12 , Fig.13 As shown; Step 3: During the search process, when a target re-indication request is received, if a re-indication target ID value is received at the same time, the corresponding GPS position and multi-layer positioning reference template are obtained according to the re-indication target ID value; if there is no re-indication target ID value, the targets corresponding to id3, id2, and id1 are re-indicated in order from the last to the first time; taking target A corresponding to id1 as an example, the re-indication method is as follows: 1. Calculate the GPS position of the center pixel of the current image based on the focal length, ground height, field of view, GPS and inertial navigation posture information of the multispectral zoom camera, use the gimbal to adjust the viewing angle of the multispectral zoom camera in real time, so that the GPS value corresponding to the center pixel of the image is the GPS position of the target A, and the image at this time is used as the marked image; if the gimbal cannot be adjusted to the abnormal situation that the GPS value corresponding to the center pixel of the image is the GPS position of the target A, the re-indication fails, and the GPS position corresponding to the center pixel of the current image is directly returned, and the GPS position corresponding to the center pixel of the current image is recalculated until the GPS value corresponding to the center pixel of the image is the GPS position of the target A, and the image at this time is used as the marked image; 2. According to the current focal length of the multi-spectral zoom camera, select and then indicate the positioning reference template of the focal length level corresponding to the target A and match it with the marked image. If the match fails, adjust the focal length of the multi-spectral zoom camera to a smaller focal length level and return to step 1. If the match succeeds, go to step 3. 3. Adjust the focal length of the multispectral zoom camera to a larger focal length level, obtain a new marked image, if the marked image fails to match the positioning reference template of the corresponding focal length level, adjust the focal length of the multispectral zoom camera to the previous focal length level, and return to step 1; if the match is successful, adjust the focal length of the multispectral zoom camera to a larger focal length level, obtain a new marked image, and then match the marked image with the positioning reference template of the corresponding focal length level; if the marked image successfully matches the positioning reference templates of all focal length levels involved in the matching, the target is indicated successfully.

[0025] The system can also set a maximum number of matches. If the number of failed matches between the marked image and the positioning reference template exceeds the set maximum number of matches, the target will be indicated as failed.

[0026] Take the positioning reference template containing 5 layers, the current focal length corresponds to the third focal length level, and the maximum number of failed matches is 4 as an example, and then the instruction method is as follows: 1. Adjust the viewing angle of the multi-spectral zoom camera so that the GPS value corresponding to the central pixel of the image is the GPS position of the target, and use the image at this time as the marked image; 2. If the positioning reference template at the third focal length level fails to match the marked image, adjust the focal length of the multispectral zoom camera to the second focal length level, return to step 1, readjust the viewing angle of the multispectral zoom camera, obtain a new marked image, and match the positioning reference template at the second focal length level with the marked image; if the positioning reference template at the third focal length level successfully matches the marked image, move the registration position in the marked image to the center of the image, and go to step 3; 3. Adjust the focal length of the multispectral zoom camera to the 4th focal length level, obtain a new marked image, and then match the marked image with the positioning reference template of the 4th focal length level; if the match fails, adjust the focal length of the multispectral zoom camera to the 3rd focal length level, return to step 1, readjust the viewing angle of the multispectral zoom camera, obtain a new marked image, and match the positioning reference template of the 3rd focal length level with the marked image; if the marked image successfully matches the positioning reference template of the 4th focal length level, move the registration position in the marked image to the image. Image center, adjust the focal length of the multispectral zoom camera to the 5th focal length level, obtain a new marked image, and then match the marked image with the positioning reference template of the 5th focal length level; if the matching fails, adjust the focal length of the multispectral zoom camera to the 4th focal length level, return to step 1, readjust the multispectral zoom camera viewing angle position, obtain a new marked image, match the positioning reference template of the 4th focal length level with the marked image; return to step 1; if the marked image matches the positioning reference template of the 5th focal length level successfully, the target is indicated successfully again. During the matching process, if the number of matching failures reaches 4 times, it will return to failure.

[0027] The GPS position of the target is obtained by the following method: Using the GPS and inertial navigation system carried by the drone, the position and attitude information of the drone is obtained, and combined with the position of the target in the image, the precise GPS coordinates of the target are calculated; the specific method is as follows: (1) Determine the flight attitude parameters of the drone: During the flight of the drone, it will be affected by factors such as wind and gravity, causing its flight attitude to change. Therefore, the drone’s attitude angles (yaw, pitch, and roll) are first measured in real time using the drone’s inertial measurement unit (IMU).

[0028] (2) Calculate the exterior orientation elements of the multispectral zoom camera: According to the flight attitude angle of the UAV and the pre-calibrated interior orientation elements of the multispectral zoom camera, calculate the exterior orientation elements of the multispectral zoom camera in the world coordinate system; X = Xc + (Hc / Z) ×(x - xp) Y = Yc + (Hc / Z) × (y - yp) Z = Hc.

[0029] Among them, (X, Y, Z) is the position coordinate of the pixel on the image in the world coordinate system, Xc, Yc, Hc is the position coordinate of the center point of the multispectral zoom camera in the world coordinate system; x, y are the position coordinates of the pixel on the image in the image coordinate system; xp, yp are the coordinates of the center point of the image; Z is the elevation of the object point.

[0030] (3) Establish the image pixel GPS coordinate conversion relationship: Through the above steps, the position coordinates of each pixel in the world coordinate system can be determined. Further, the position coordinates can be converted into standard GPS coordinates to obtain the GPS coordinates corresponding to each pixel. The conversion formula is as follows: Latitude = arctan[(Z - Hc) / sqrt(X^2 + Y^2)] Longitude = arctan(Y / X).

[0031] Among them, Latitude is latitude and Longitude is longitude.

[0032] The positioning reference template and the marked image may be matched using an existing feature matching algorithm, a histogram matching algorithm, a template matching algorithm, a deep learning-based matching algorithm, or a coefficient normalization algorithm (COEFF NORMED).

[0033] Feature matching algorithm: Image matching is achieved by detecting feature points in the image and calculating feature descriptors. Commonly used feature matching algorithms include SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (OrientedFAST and Rotated BRIEF), etc.

[0034] Histogram matching algorithm: Image matching is achieved by calculating the histogram features of the image. Commonly used algorithms include histogram similarity matching, histogram-based color matching, etc.

[0035] Based on template matching algorithm: Matching is achieved by sliding the image to be matched on the template image and calculating the similarity between the two images. Commonly used template matching algorithms include SSD (Sum of Squared Differences) and NCC (Normalized Cross Correlation).

[0036] Matching algorithm based on deep learning: Use convolutional neural network (CNN), recurrent neural network (RNN) and other methods in deep learning technology to extract and match image features.

[0037] Coefficient normalization algorithm: Match the relative value of the template image to its mean with the relative value of the image to be matched to its mean, 1 indicates a perfect match, -1 indicates a poor match, and 0 indicates no correlation.

[0038] The present invention uses a multi-sensor fusion target optical detection method to improve the accuracy and robustness of detection. Target recognition and positioning uses a multispectral zoom camera to obtain multispectral image data of the target, and performs denoising, correction, alignment and other processing on the collected data to improve the quality of the image data. A dynamic target recognition method is used to control the pan / tilt and multispectral zoom camera in real time. The image sensor has high sensitivity and a wide dynamic range, and can obtain clear images in complex environments; the collected original image is pre-processed by filtering, correction, alignment and other pre-processing to eliminate noise and distortion, and prepare for subsequent feature extraction.

Claims

1. A multi-modal real-time target positioning and re-indication method, characterized in that The method is as follows: Step 1: Perform target recognition and positioning on the original images collected in real time by the multi-spectral zoom camera carried by the drone, assign an ID value to the recognized target and record its GPS location; Step 2: dynamically adjust the focal length of the multispectral zoom camera, and use the target area image captured within the adjustable focal length range of each segment as the positioning reference template of the corresponding focal length level, so as to generate a multi-layer positioning reference template containing focal length information; Step 3: When receiving a target re-indication request, first obtain the GPS position and multi-layer positioning reference template corresponding to the re-indication target; If the marked image successfully matches the positioning reference templates of all focal length levels involved in the matching, the target re-indication is successful; the marked image is an image when the GPS value corresponding to the central pixel of the image is the GPS position of the re-indicated target.

2. The method for real-time target positioning and re-indication based on multi-modal according to claim 1, characterized in that: In the step 2, the gimbal posture is first adjusted so that the target in the image is at or close to the center, and then the image of the target area is captured within the adjustable focal length range of each segment as a positioning reference template for the corresponding focal length level.

3. The method for real-time target positioning and re-indication based on multi-modal according to claim 2, characterized in that: In the step 2, the image of the target area is centered on the target coverage frame and is proportionally enlarged by 1.2-1.5 times.

4. The method for real-time target positioning and re-indication based on multi-modal according to claim 1, characterized in that: In the step one, multiple targets are identified, one of which is located first, an ID value is assigned to it, and its GPS position is recorded, and then a multi-layer positioning reference template corresponding to the target is generated through step two; similarly, other identified targets are located in turn, ID values ​​are assigned to them, their GPS positions are recorded, and corresponding multi-layer positioning reference templates are generated.

5. The method for real-time target positioning and re-indication based on multi-modal according to claim 1, characterized in that: In the step three, when a target re-indication request is received, if a re-indication target ID value is received at the same time, the corresponding GPS information and multi-layer positioning reference template are obtained according to the re-indication target ID value.

6. The method for real-time target positioning and re-indication based on multi-modal according to claim 1, characterized in that: In the step 3, when a target re-indication request is received, if a re-indication target ID value is not received, the targets corresponding to the ID values ​​are re-indicated in order from the latest to the earliest in time.

7. The method for real-time target positioning and re-indication based on multi-modal according to claim 1, characterized in that: In step 3, the method for matching the marked image with the positioning reference template is as follows: 1) Adjust the viewing angle of the multispectral zoom camera so that the GPS value corresponding to the central pixel of the image indicates the GPS position of the target, and use the image at this time as the marked image; 2) According to the current focal length of the multispectral zoom camera, select and then indicate the positioning reference template of the focal length level corresponding to the target and match it with the marked image; if the match fails, adjust the focal length of the multispectral zoom camera to a smaller focal length level and return to step 1; if the match is successful, move the registration position in the marked image to the center of the image and go to step 3; 3) Adjust the focal length of the multispectral zoom camera to a larger focal length level, obtain a new marked image, if the marked image fails to match the positioning reference template of the corresponding focal length level, adjust the focal length of the multispectral zoom camera to the previous focal length level, and return to step 1; if the match is successful, adjust the focal length of the multispectral zoom camera to a larger focal length level, obtain a new marked image, and then match the marked image with the positioning reference template of the corresponding focal length level; if the marked image successfully matches the positioning reference templates of all focal length levels involved in the matching, the target is indicated successfully again.

8. The method for real-time target positioning and re-indication based on multi-modal according to claim 7, characterized in that: If the number of failed matches between the marker image and the positioning reference template exceeds the set maximum number of matches, the target will then indicate failure.

9. The method for real-time target positioning and re-indication based on multi-modal according to claim 1, characterized in that The marked image and the positioning reference template are matched using a feature matching algorithm, a histogram matching algorithm, a template-based matching algorithm, a deep learning-based matching algorithm or a coefficient normalization algorithm.

Images