Ship precise positioning method and system based on three-dimensional depth feature extraction

Abstract

The application provides a ship precise positioning method and system based on three-dimensional depth feature extraction, comprising the following steps: S1, based on the optical stereo imaging principle, combined with the photographic pose data, a plurality of view images imaged at different angles and different distances are acquired; S2, a region of interest frame is identified in the acquired images, the ship target in the images is framed, and key point identification is performed on the target in the region of interest; S3, the key points containing semantics are identified, and the key points identified in the images photographed at different angles are paired according to semantic labels; S4, the target is classified and identified by using the three-dimensional space coordinate information of the key points, the real model in the database is virtually arranged into the body coordinate system according to the position of the key points in the space after the ship model is identified, and the false alarm and the missed detection of the ship target caused by noise interference and environmental interference are reduced; and the application can capture more feature points, and the identification accuracy is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of transportation, in particular to a ship precise positioning method and system based on three-dimensional deep feature extraction. BACKGROUND

[0002] Traditional ship target detection methods are mostly based on manually designed features, which have high detection efficiency for ship targets on calm sea surfaces in the open sea, but have high false alarm rates and low accuracy for ship target detection in complex background environments, such as sea wave interference, similar shape object interference, and shore facilities, containers, vehicles, etc. in high-resolution images in near-shore scenes.

[0003] In recent years, with the key breakthroughs in the field of artificial intelligence in the direction of deep learning, deep learning methods have also been widely applied to ship target detection and identification. Compared with traditional ship detection and identification methods, the nonlinear fitting capability of the ship target detection and identification method based on deep learning is stronger, and it is more suitable for ship target detection and identification in complex scenes. Deep learning models can also have better detection and identification capabilities for targets through training on a large number of sample data, and the models are easy to maintain. When subsequent samples continue to accumulate, the detection and identification effect can be improved by retraining the model without redesigning the model.

[0004] Patent document CN105180943A discloses a ship positioning system, comprising: a position information acquisition unit for acquiring multiple position information of a ship from multiple different positioning ends; a sorting unit for sorting the acquired multiple position information according to priority, and sending the multiple position information with priority greater than a set threshold to a fusion unit; a fusion unit for fusing the multiple position information with priority greater than the set threshold to obtain a unique ship latitude and longitude coordinate; and a monitoring unit for acquiring ship monitoring images according to the ship latitude and longitude coordinate. However, this invention does not solve the problem of the adaptability of detection and identification algorithms to scale and rotation. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a ship precise positioning method and system based on three-dimensional deep feature extraction.

[0006] According to the ship precise positioning method based on three-dimensional deep feature extraction provided by the present application, the method comprises:

[0007] Step S1: based on the principle of optical stereo imaging, combined with photographic pose data, multiple view images imaged at different angles and different distances are acquired;

[0008] Step S2: a region of interest frame is identified in the acquired image, the ship target in the image is framed, and key point identification is performed on the target in the region of interest.

[0009] Step S3: identifying the key points containing semantics, the key points identified in the images taken at different angles are paired according to semantic labels;

[0010] Step S4: using the three-dimensional spatial coordinate information of the key points to classify and identify the target, identifying the ship model, virtually placing the real model in the database into the body coordinate system according to the position of the key points in space.

[0011] Preferably, in the step S1:

[0012] When the target meets multiple aircraft trajectory conditions, multiple aircraft or multiple aircraft trajectory mixed cooperative search is adopted according to tactical needs;

[0013] When multiple aircrafts cooperatively investigate, they communicate with each other, use the information obtained by multiple aircrafts to jointly solve the three-dimensional information of the target for identification and key point positioning; communication adopts a broadcast mode, each aircraft broadcasts the information obtained by itself to other aircrafts, and receives the information sent by other aircrafts, each aircraft obtains complete data, and independently solves after receiving all data;

[0014] Each aircraft takes a total of three times when positioning the end section, the first time of photography, the aircraft is farthest from the target, the field of view is largest, and the target identification algorithm can determine the search target from multiple targets;

[0015] After the first photography, the flight control system adjusts the flight direction according to the positioning result, and the target appears in the center area of the image during subsequent photography, which narrows the search range of the target region of interest frame and shortens the data processing time; at the same time, when the accuracy of the range of the region of interest frame is greater than the preset standard, the actual camera imaging switches to the region of interest imaging mode, which only images within the region of interest.

[0016] Preferably, feature point identification and calibration are performed in advance according to the ship model in the database, and each aircraft takes a total of three times when searching the end section, after each photography, the following steps are performed:

[0017] Identify the region of interest frame in the obtained image, frame the ship target in the image, and identify the key points containing semantics in the region of interest;

[0018] The key points identified in the images taken at different angles contain three-dimensional features, and are paired according to semantic labels;

[0019] The photograph coordinates of the identified key points and the photography pose parameters recorded by the navigation device on the missile during photography are sent to all other aircrafts, and the data sent by other aircrafts is received at the same time, the photography pose includes exposure position and photography attitude;

[0020] Pose data solution: O is the lens center of the camera, A is the target point in the scene space, within the exposure time Δt, the target point moves to B:

[0021]

[0022] Where θ is the angle between the imaging principal axis and the vertical line, v is the aircraft speed, and t is the flight time;

[0023] f is the focal length of the camera, s is the distance between the camera and the target, s = H / cosθ, H is the flight height;

[0024] α is the field of view angle, which is the angle between the half image size and the focal length. The calculation formula of the field of view angle is:

[0025]

[0026] Where p s is the pixel size, and w is the image size;

[0027] In the horizontal direction:

[0028] Δx is the image shift, the movement of the flying body causes the target point A to move from the image point c to b on the image plane, and the distance on the image plane is the image shift Δx to be solved;

[0029] Since Therefore:

[0030]

[0031] In the vertical direction:

[0032] Δx is the image shift, the movement of the flying body causes the target point A to move from the image point a to b on the image plane, and the distance on the image plane is the image shift Δx to be solved;

[0033] Since Therefore

[0034]

[0035]

[0036] Where v n is v n is the flight speed of the aircraft n, n = 5;

[0037] The key point images collected by each flight of the different aircraft in the cooperative search form observations of the same image points of the same space point in different images.

[0038] Preferably, in the step S3:

[0039] Convert all the received photographic pose data to the body coordinate system, take the aircraft body coordinate system as the reference coordinate system for joint solution, and take the key point image coordinates identified by itself and the key point image coordinates received from other aircraft as the image point observation data required for joint solution;

[0040] The photographic pose of the aircraft itself and the photographic pose of other aircraft are taken as control conditions, and the three-dimensional space coordinates of all key points in the aircraft body coordinate system and three-dimensional feature information are solved through joint block adjustment technology, the target position is identified, and the direction vector of the strike is calculated.

[0041] Preferably, in the step S4:

[0042] After determining the type of the ship by using the three-dimensional information of the key points identified on the image, the three-dimensional model of the ship is placed into the reference coordinate system of the aircraft according to the space coordinates of the key points solved by the block adjustment, and any specified position on the ship is positioned.

[0043] According to the ship precise positioning system based on three-dimensional deep feature extraction provided by the application, comprising:

[0044] Module M1: based on the principle of optical stereo imaging, combined with photographic pose data, a multi-view image imaged at different angles and different distances is obtained;

[0045] Module M2: a region of interest frame is identified in the obtained image, the ship target in the image is framed, and key point identification is performed on the target in the region of interest;

[0046] Module M3: the key points containing semantics are identified, and the key points identified in the images photographed at different angles are paired according to semantic labels;

[0047] Module M4: the target is classified and identified by using the three-dimensional space coordinate information of the key points, the real model in the database is virtually placed into the body coordinate system according to the position of the key points in space after the ship model is identified.

[0048] Preferably, in the module M1:

[0049] When the target meets multiple aircraft trajectory conditions, multiple aircrafts or multiple aircraft trajectory mixed cooperative search are adopted according to tactical needs;

[0050] The multiple aircrafts communicate with each other when cooperatively detecting, and the three-dimensional information of the target is obtained by solving the information obtained by the multiple aircrafts to identify and locate the key points; the communication adopts a broadcast mode, each aircraft broadcasts the information obtained by itself to other aircrafts, and receives the information sent by other aircrafts, each aircraft obtains complete data, and solves independently after receiving all the data;

[0051] Each aircraft takes a photograph three times in the final positioning stage, the first photograph is taken when the aircraft is farthest from the target, the field of view is largest, and the target identification algorithm can determine the search target from multiple targets;

[0052] After the first photograph, the flight control system adjusts the flight direction according to the positioning result, and the target appears in the center area of the image during subsequent photography, which reduces the search range of the target area of interest and shortens the data processing time; at the same time, when the accuracy of the range of the area of interest is greater than the preset standard, the actual camera imaging switches to the imaging mode of the area of interest, and only images within the range of the area of interest.

[0053] Preferably, the feature points are identified and calibrated in advance according to the ship model in the database, and each aircraft takes a photograph three times in the final search stage, and after each photograph, the following steps are performed:

[0054] Identify the area of interest frame in the obtained image, frame the ship target in the image, and identify the key points containing semantics in the area of interest;

[0055] The key points identified in the images taken at different angles contain three-dimensional features, and are paired according to semantic labels;

[0056] The photograph coordinates of the identified key points and the photograph pose parameters recorded by the navigation device on the missile during photography are sent to all other aircrafts, and data sent by other aircrafts is received, and the photograph pose includes the exposure position and the photograph attitude;

[0057] Pose data solving: O is the center of the camera lens, and A is the target point in the scene space, within the exposure time Δt, the target point moves to B:

[0058]

[0059] Wherein, θ is the included angle between the imaging principal axis and the plumb line, v is the speed of the aircraft, and t is the flight time;

[0060] f is the focal length of the camera, s is the distance between the camera and the target, s = H / cosθ, and H is the flight height;

[0061] α is the field of view angle, which is the included angle formed by the half image width and the focal length, and the calculation formula of the field of view angle is:

[0062]

[0063] where p s is the pixel size, w is the image width;

[0064] In horizontal direction:

[0065] Δx is the image shift, the motion of the flying body causes the target point A to move from image point c to b on the image plane, the distance on the image plane is Δx is the image shift to be solved;

[0066] Since Therefore:

[0067]

[0068] In vertical direction:

[0069] Δx is the image shift, the motion of the flying body causes the target point A to move from image point a to b on the image plane, the distance on the image plane is Δx is the image shift to be solved;

[0070] Since Therefore

[0071]

[0072]

[0073] where v n v n is the flight speed of the aircraft n, n = 5;

[0074] The key point images taken by different aircrafts in each cooperative search are composed of the same image points of the same space point in different images.

[0075] Preferably, in the module M3:

[0076] All received photographic pose data are converted to the body coordinate system, and the body coordinate system of the aircraft is used as the reference coordinate system for joint calculation. The key point image coordinates identified by itself and the key point image coordinates received from other aircrafts are used as the image point observation data required for joint calculation.

[0077] The photographic pose of the aircraft itself and the photographic pose of other aircrafts are used as control conditions, and the three-dimensional space coordinates of all key points in the body coordinate system of the aircraft and three-dimensional feature information are calculated through joint block adjustment technology. The target position is identified and the direction vector of the attack is calculated.

[0078] Preferably, in the module M4:

[0079] After determining the ship type by using the three-dimensional information of the key points recognized on the image, the three-dimensional model of the ship is placed into the reference coordinate system of the aircraft according to the key point space coordinates solved by the block adjustment, and a specified position on the ship is positioned.

[0080] Compared with the prior art, the present application has the following beneficial effects:

[0081] 1. The present application reduces false alarms and missed detections of ship targets caused by noise interference and environmental interference;

[0082] 2. The present application can capture more feature points and greatly improve the recognition accuracy;

[0083] 3. Different shooting angles, different shooting distances, and different ship orientations make the targets in the image have scale variability and rotation variability, and the present application solves the adaptability problem of detection and recognition algorithms to scale and rotation;

[0084] 4. The present application does not need to classify features such as texture and color of the image, and does not need to perform deep learning on large data, and the real-time detection performance is guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0085] Other features, objects and advantages of the present application will become more apparent after reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0086] Figure 1 The present application is a method flowchart;

[0087] Figure 2 The present application is a visual algorithm flowchart;

[0088] Figure 3 The present application is a high-low composite aircraft trajectory mode diagram;

[0089] Figure 4 The present application is a high-low composite aircraft trajectory attack diagram;

[0090] Figure 5 The present application is an optical camera exposure time diagram during the final search;

[0091] Figure 6 The present application is a key point matching simulation diagram; DETAILED DESCRIPTION

[0092] The present application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These all belong to the protection scope of the present application.

[0093] Embodiment 1

[0094] The present application relates to a depth feature recognition positioning method under a three-dimensional model, which is suitable for ship target detection and recognition. It is mainly used for realizing the recognition and positioning function of the ship in the model. In the scenes of long-distance large-scale anti-terrorism operations, long-distance fishing boat search, search and rescue, etc., the multi-view vision method is used for feature recognition and extraction, and then target positioning is realized.

[0095] Firstly, based on the optical stereo imaging principle, combined with the photographic pose data, that is, using multi-view images imaged at different angles and different distances, and obtaining the region of interest of the image, then identifying the key points in the region of interest, and directly pairing the key points identified in the images photographed at different angles according to the semantic label.

[0096] Multi-view vision constitutes the homonymous point observation of the same space point on different images, converts all photographic pose data to the body coordinate system, and the key point image coordinates identified by itself and the key point image coordinates received from other cameras are used as the point observation data required for joint solution. Through joint block adjustment technology, the three-dimensional space coordinates of all key points in the unmanned aerial vehicle body coordinate system are solved.

[0097] Finally, the three-dimensional space coordinate information of the key points is used to classify and identify the target, and after identifying the ship model, the real model in the database can be virtually placed into the body coordinate system according to the position of the key points in space, and the direction vector of the accurate attack is calculated according to the key points on the target.

[0098] According to the ship precise positioning method based on three-dimensional depth feature extraction provided by the present application, as shown in Figures 1-6 , comprising:

[0099] Step S1: Based on the optical stereo imaging principle, combined with the photographic pose data, multi-view images imaged at different angles and different distances are obtained;

[0100] Specifically, in the step S1:

[0101] When the target meets the conditions of multiple aircraft trajectories, multiple aircrafts or multiple aircraft trajectory mixed cooperative search are adopted according to the tactical needs;

[0102] When multiple aircrafts cooperate in reconnaissance, they communicate with each other, use the information obtained by multiple aircrafts to jointly solve the three-dimensional information of the target for identification and key point positioning; communication adopts a broadcast mode, each aircraft broadcasts the information obtained by itself to other aircrafts, and receives the information sent by other aircrafts, each aircraft obtains complete data, and independently solves after receiving all data;

[0103] Each aircraft takes three photos in the final positioning stage, the first photo is taken when the aircraft is farthest from the target, the field of view is largest, and the target recognition algorithm can determine the search target from multiple targets;

[0104] After the first photo, the flight control system adjusts the flight direction according to the positioning results, and the target appears in the center area of the image in subsequent photos, reducing the search range of the target region of interest and shortening the data processing time. At the same time, when the accuracy of the region of interest frame is greater than the preset standard, the actual camera imaging switches to the region of interest imaging mode, and only images within the region of interest.

[0105] Specifically, the feature points are identified and calibrated in advance according to the ship model in the database. During the search, each aircraft takes three photos in the final search stage, and after each photo, the following steps are performed:

[0106] Identify the region of interest frame in the acquired image, frame the ship target in the image, and identify the key points in the region of interest that contain semantics;

[0107] The key points identified in the images taken from different angles contain three-dimensional features, and are paired according to semantic labels;

[0108] Send the image coordinates of the identified key points and the photo pose parameters recorded by the navigation device on the missile during photography to all other aircraft, and receive data from other aircraft. The photo pose includes the exposure position and the photography attitude;

[0109] Pose data solution: O is the center of the camera lens, A is the target point in the scene space, and within the exposure time Δt, the target point moves to B:

[0110]

[0111] Where θ is the angle between the imaging principal axis and the vertical line, v is the speed of the aircraft, and t is the flight time;

[0112] f is the focal length of the camera, s is the distance between the camera and the target, s = H / cosθ, H is the flight height;

[0113] α is the field of view angle, which is the angle between the half image size and the focal length. The calculation formula of the field of view angle is:

[0114]

[0115] Where p s is the pixel size, and w is the image width;

[0116] In the horizontal direction:

[0117] Δx is the image shift, the movement of the flying body causes the target point A to move from a to b on the image plane, the distance on the image plane Δx is the image shift, the movement of the flying body causes the target point A to move from a to b on the image plane, the distance on the image plane

[0118] Since Therefore

[0119]

[0120] In the vertical direction:

[0121] Δx is the image shift, the movement of the flying body causes the target point A to move from a to b on the image plane, the distance on the image plane Δx is the image shift, the movement of the flying body causes the target point A to move from a to b on the image plane, the distance on the image plane

[0122] Since Therefore

[0123]

[0124]

[0125] Where, v n v is the flight speed of the aircraft n, n = 5 n v is the flight speed of the aircraft n, n = 5

[0126] The different aircrafts in the group of cooperative search form the same space point on the same image point observation in different images by taking pictures of the key points collected each time.

[0127] Step S2: identify the region of interest frame in the obtained image, frame the ship target in the image, and identify the key points in the target of interest region;

[0128] Step S3: identify the key points containing semantics, and pair the key points identified in the images taken at different angles according to semantic labels;

[0129] Specifically, in the step S3:

[0130] Convert all received photographic pose data to the body coordinate system, use the aircraft body coordinate system as the reference coordinate system for joint calculation, and use the key point image coordinates identified by itself and the key point image coordinates received from other aircraft as the image point observation data required for joint calculation;

[0131] The photographic pose of the aircraft itself and the photographic pose of other aircraft are used as control conditions, and all key points in the aircraft body coordinate system are calculated by joint block adjustment technology to obtain three-dimensional space coordinates and three-dimensional feature information, and the target position is identified and the direction vector of the attack is calculated.

[0132] Step S4: using the three-dimensional spatial coordinate information of the key points to classify and identify the target, identifying the ship model, and virtually placing the real model in the database into the body coordinate system according to the position of the key points in space.

[0133] Specifically, in the step S4:

[0134] After determining the ship type by using the three-dimensional information of the identified key points on the image, the three-dimensional model of the ship is placed into the reference coordinate system of the aircraft according to the key point spatial coordinates calculated by the block adjustment, and the position of any specified position on the ship is located.

[0135] Embodiment 2:

[0136] Embodiment 2 is a preferred example of Embodiment 1, which is used to more specifically illustrate the present application.

[0137] The present application also provides a ship precision positioning system based on three-dimensional depth feature extraction, which can be realized by executing the flow steps of the ship precision positioning method based on three-dimensional depth feature extraction, that is, the ship precision positioning method based on three-dimensional depth feature extraction can be understood as the preferred embodiment of the ship precision positioning system based on three-dimensional depth feature extraction by those skilled in the art.

[0138] According to the ship precision positioning system based on three-dimensional depth feature extraction provided by the present application, the ship precision positioning system based on three-dimensional depth feature extraction comprises:

[0139] Module M1: based on the principle of optical stereo imaging, combined with photographic pose data, a plurality of view images imaged at different angles and different distances are obtained;

[0140] Specifically, in the module M1:

[0141] When the target meets multiple aircraft trajectory conditions, multiple aircrafts or multiple aircraft trajectory mixed cooperative search are adopted according to tactical needs;

[0142] When multiple aircrafts cooperatively investigate, they communicate with each other, and the three-dimensional information of the target obtained by using multiple aircrafts is jointly calculated to identify and locate the key points; the communication adopts a broadcast mode, each aircraft broadcasts the information obtained by itself to other aircrafts, and receives the information sent by other aircrafts, each aircraft obtains complete data, and independently calculates after receiving all the data;

[0143] Each aircraft photographs three times in total when positioning the end segment, the first time of photographing, the aircraft is farthest from the target, the field of view is largest, and the target identification algorithm can determine the search target from multiple targets;

[0144] The flight control system adjusts the flight direction according to the positioning result after the first photographing, and the target appears in the center area of the image during subsequent photographing, thereby reducing the search range of the target region of interest and shortening the data processing time; meanwhile, when the accuracy of the region of interest frame is greater than the preset standard, the actual camera imaging is switched to the region of interest imaging mode, and only the region of interest range is imaged.

[0145] Specifically, the feature point recognition and calibration are performed in advance according to the ship model in the database, and during the search, each aircraft photographs three times in the final search, and after each photographing, the following steps are performed:

[0146] The region of interest frame is identified in the obtained image, and the ship target in the image is framed, and the key points containing semantics in the region of interest are identified;

[0147] The key points identified in the images photographed from different angles contain three-dimensional features, and are matched according to the semantic labels;

[0148] The photograph coordinates of the identified key points and the photographing pose parameters recorded by the navigation device on the missile during photographing are sent to all other aircrafts, and the data sent by other aircrafts is received, and the photographing pose includes the exposure position and the photographing attitude;

[0149] Pose data solving: O is the lens center of the camera, and A is the target point in the scene space, and within the exposure time Δt, the target point moves to B:

[0150]

[0151] Wherein, θ is the included angle between the imaging main optical axis and the vertical line, v is the speed of the aircraft, and t is the flight time;

[0152] f is the focal length of the camera, s is the distance between the camera and the target, s = H / cosθ, and H is the flight height;

[0153] α is the field of view angle, which is the included angle formed by the half image size and the focal length, and the calculation formula of the field of view angle is:

[0154]

[0155] Wherein, p s is the pixel size, and w is the image width;

[0156] In the horizontal direction:

[0157] Δx is the image shift, the movement of the flight body causes the target point A to move from c to b on the image plane, and the distance on the image plane is the image shift Δx to be solved;

[0158] Since Therefore:

[0159]

[0160] In the vertical direction:

[0161] Δx is the image motion, the movement of the flying body causes the target point A to move to b in the image plane, the distance in the image plane is the image motion Δx to be solved;

[0162] Since Therefore

[0163]

[0164]

[0165] Where, v n is v n is the flight speed of the aircraft n, n=5;

[0166] The different aircraft in this group of cooperative search form the same space point in different images, and the same image point observation is observed.

[0167] Module M2: identify the region of interest frame in the obtained image, frame the ship target in the image, and identify the key points in the region of interest;

[0168] Module M3: identify the key points containing semantics, and pair the key points identified in the images taken at different angles according to semantic labels;

[0169] Specifically, in the module M3:

[0170] Convert all received photographic pose data to the body coordinate system, use the aircraft body coordinate system as the reference coordinate system for joint calculation, and use the key point image coordinates identified by itself and the key point image coordinates received from other aircraft as the image point observation data required for joint calculation;

[0171] The photographic pose of the aircraft itself and the photographic pose of other aircraft are used as control conditions, and all key points in the aircraft body coordinate system are calculated by joint block adjustment technology. Three-dimensional space coordinates and three-dimensional feature information are calculated, the target position is identified, and the direction vector of the attack is calculated.

[0172] Module M4: use the three-dimensional space coordinate information of the key points to classify and identify the target, identify the ship model, and virtually place the real model in the database into the body coordinate system according to the position of the key points in space.

[0173] Specifically, in the module M4:

[0174] After determining the type of the ship using the three-dimensional information of the key points identified on the image, the three-dimensional model of the ship is placed into the reference coordinate system of the aircraft according to the spatial coordinates of the key points solved by the block adjustment, and any specified position on the ship is located.

[0175] Embodiment 3:

[0176] Embodiment 3 is a preferred example of Embodiment 1, which is used to more specifically illustrate the present application.

[0177] The ship precise positioning method based on three-dimensional depth feature extraction includes the following steps:

[0178] Step 1: When the target meets multiple aircraft trajectory conditions, multiple unmanned aircraft or multiple aircraft trajectory mixed cooperative search can be used according to tactical needs;

[0179] Step 2: Identify the ROI (Region of Interest) frame in the obtained image, frame the ship target in the image, and then identify the key points of the target in the ROI;

[0180] Step 3: Identify the key points containing semantics, and the key points identified in images taken at different angles can be directly paired according to semantic labels;

[0181] Step 4: Take the unmanned aerial vehicle body coordinate system as the reference coordinate system for joint solution, and the key point image coordinates identified by itself and the key point image coordinates received from other unmanned aerial vehicles are used as the image point observation data required for joint solution.

[0182] Step 5: Use the three-dimensional spatial coordinate information of the key points to classify and identify the target, and after identifying the ship model, the real model in the database can be virtually placed into the body coordinate system according to the spatial position of the key points.

[0183] When multiple aircrafts cooperate in reconnaissance, they need to communicate with each other in order to use the information obtained by multiple aircrafts to jointly solve the three-dimensional information of the target for identification and key point positioning. Communication uses a broadcast mode, that is, each unmanned aerial vehicle broadcasts the information obtained by itself to other unmanned aerial vehicles, and also receives the information sent by other unmanned aerial vehicles. In this way, each unmanned aerial vehicle can obtain complete data, and after receiving all the data, each unmanned aerial vehicle independently solves. Even if an unmanned aerial vehicle is intercepted in the middle, the unmanned aerial vehicle still independently solves based on all the data that can be received during the search flight.

[0184] Each UAV takes three photos in total when positioning the final stage, the first photo is taken when the UAV is farthest from the target, at this time the field of view is the largest, so as to capture as many targets as possible in the imaging range, so that the target identification algorithm can determine the search target from multiple targets;

[0185] After the first photo, the flight control system adjusts the flight direction according to the positioning result, so that the target should appear in the center area of the image in the subsequent photos, so as to narrow the search range of the identified target ROI frame and shorten the data processing time; at the same time, once the ROI frame range is accurately determined, the actual camera imaging can be switched to the ROI imaging mode, i.e. only imaging in the ROI range;

[0186] After determining the type of the ship using the three-dimensional information of the key points identified on the image, the three-dimensional model of the ship can be directly placed into the reference coordinate system of the UAV according to the spatial coordinates of the key points solved by the aforementioned area network adjustment, so that the specified position on the ship can be positioned with high precision.

[0187] High-low UAV trajectory combination mode, better combines the advantages of high and low UAV trajectories, adopts a cooperative search scheme as shown in Figure 2 , 5 UAVs approach the target from 5 different angles, and the final flight trajectories of the 5 UAVs adopt a combination of 3 high and 2 low.

[0188] Each UAV takes three photos in total when searching the final stage, and uses a method based on visual saliency to obtain the region of interest (ROI) in the obtained image, and the ROI frame frames the ship target in the image; then the key points of the target in the ROI are identified, and the key points containing semantics are identified. The key points containing semantic information pre-marked are directly matched with the key points identified in actual application according to the semantic label.

[0189] The image coordinates of the identified key points and the photo pose (including exposure position and photo attitude) parameters recorded by the navigation equipment on the UAV when taking photos are sent to all other UAVs, and the data sent by other UAVs is received.

[0190] The key points identified on different UAVs can be directly paired according to semantic labels to form homonymous image point observations of the same space point on different images, all received photographic pose data is converted to the body coordinate system, the body coordinate system of the UAV is used as the reference coordinate system for joint calculation, the key point image coordinates identified by the UAV and the key point image coordinates received from other UAVs (each photographic data of each UAV) are used as image point observation data required for joint calculation, the photographic pose of the UAV and the photographic pose of other UAVs are used as control conditions, and all key points in the body coordinate system of the UAV are calculated by joint block adjustment technology.

[0191] The target is classified and identified using the three-dimensional space coordinate information of the key points, after identifying the model of the ship, the real model in the database can be virtually placed in the body coordinate system according to the position of the key points in space, and finally the direction vector of the accurate attack is calculated for the key points on the target.

[0192] According to the existing model, the feature points are identified and calibrated in advance, and during the search, each UAV takes three photographs in total during the final search, and after each photograph, the following algorithm steps are executed:

[0193] 1. Identify the ROI (Region of Interest) frame in the obtained image, frame the ship target in the image, and then identify the key points containing semantics in the ROI;

[0194] 2. The key points identified in the images taken from different angles contain three-dimensional features, which can be directly paired according to semantic labels;

[0195] 3. The image coordinates of the identified key points and the photographic pose (including exposure position and photographic attitude) parameters recorded by the navigation device on the missile during photography are sent to all other UAVs, and data from other UAVs is received.

[0196] Pose data calculation: O is the lens center of the camera, A is the target point in the scene space, considering that the motion between the camera and the target is relative, it can be assumed that the camera is stationary, and the target point A moves towards the camera within the exposure time Δt, and the target point moves to point B. That is Where θ is the included angle between the imaging principal axis and the plumb line (as shown in Figure 1 ).

[0197] f is the focal length of the camera, generally in millimeters.

[0198] s is the distance between the camera and the target. According to Figure 1 , s = H / cosθ, H is the altitude.

[0199] a is the field of view, i.e. the angle between the half image size and the focal length. Assuming the image size is 1280x1024 pixels and the pixel size is 14.1 μm, the field of view is calculated as follows where p s is the pixel size, and w is the image size (the wide side, i.e. 1280 pixels, is used in the calculation).

[0200] In the horizontal direction:

[0201] Δx is the image shift, and the distance on the image plane from the target point A to the point b is caused by the movement of the flying body, so that the image point c moves to b. The distance on the image plane is the image shift Δx to be solved.

[0202] Since Therefore:

[0203]

[0204] In the vertical direction:

[0205] Δx is the image shift, and the distance on the image plane from the target point A to the point b is caused by the movement of the flying body, so that the image point c moves to b. The distance on the image plane is the image shift Δx to be solved.

[0206] Since Therefore

[0207]

[0208]

[0209] The key point images taken by each of the different unmanned aerial vehicles in the cooperative search are thus composed of the same-named image points of the same space point in different images;

[0210] 4. All the received photographic pose data are converted to the body coordinate system, and the body coordinate system of the unmanned aerial vehicle is used as the reference coordinate system for joint calculation. The key point image coordinates identified by the unmanned aerial vehicle and the key point image coordinates received from other unmanned aerial vehicles are used as the image point observation data required for joint calculation;

[0211] 5. The photographic pose of the unmanned aerial vehicle and the photographic pose of other unmanned aerial vehicles are used as control conditions, and the three-dimensional space coordinates of all the key points in the body coordinate system of the unmanned aerial vehicle and the three-dimensional feature information are calculated through joint block adjustment technology, so as to identify the target position and calculate the direction vector of the accurate attack.

[0212] Those skilled in the art know that, in addition to implementing the system, device and each module thereof provided by the present application in the form of pure computer readable program code, the same program can also be implemented in the form of logic gate, switch, special integrated circuit, programmable logic controller and embedded microcontroller, etc. by logically programming the method steps. Therefore, the system, device and each module thereof provided by the present application can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures in the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing methods and structures in the hardware component.

[0213] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various changes or modifications can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily without conflict.

Claims

1. A ship precise positioning method based on three-dimensional depth feature extraction, characterized in that, Comprise: Step S1: based on the principle of optical stereo imaging, combined with photographic pose data, obtain multi-view images imaged at different angles and different distances; Step S2: identify the region of interest frame in the obtained image, frame the ship target in the image, and identify the key points in the region of interest; Step S3: identify the key points containing semantics, and pair the key points identified in the images taken at different angles according to semantic labels; Step S4: classify and identify the target using the three-dimensional spatial coordinate information of the key points, identify the ship model, and virtually place the real model in the database into the body coordinate system according to the position of the key points in space; In the step S1: When the target meets multiple aircraft trajectory conditions, multiple aircraft or multiple aircraft trajectory mixed cooperation search is adopted according to tactical needs; Multiple aircrafts communicate with each other when they cooperate in investigation, and the three-dimensional information of the target is obtained by jointly solving the information obtained by multiple aircrafts for identification and key point positioning; Communication adopts broadcast mode, each aircraft broadcasts the information obtained by itself to other aircrafts, and receives the information sent by other aircrafts, each aircraft obtains complete data, and independently solves after receiving all data; Each aircraft takes a total of three times at the end of positioning, the first time, the aircraft is farthest from the target, the field of view is largest, and the target identification algorithm can determine the search target from multiple targets; After the first photography, the flight control system adjusts the flight direction according to the positioning result, the target appears in the center area of the image during subsequent photography, the search range of the target region of interest frame is reduced, and the data processing time is shortened; At the same time, when the accuracy of the range of the region of interest frame is greater than the preset standard, the actual camera imaging is switched to the imaging mode of the region of interest, and only the region of interest is imaged; According to the feature point recognition and calibration of the ship model in the database, each aircraft takes a total of three times at the end of search, and after each photography, the following steps are executed: Identify the region of interest frame in the obtained image, frame the ship target in the image, and identify the key points containing semantics in the region of interest; The key points identified in the images taken at different angles contain three-dimensional features, and are paired according to semantic labels; Send the photograph coordinates of the identified key points and the photographic pose parameters recorded by the navigation device on the missile during photography to all other aircrafts, and receive the data sent by other aircrafts at the same time, the photographic pose includes exposure position and photographic attitude; Pose data solving: the lens center of the point camera, A point is a target point in the scene space, which moves relatively to the camera within the exposure time A point: wherein is the angle between the imaging principal axis and the plumb line, aircraft speed, is the flight time; is the focal length of the camera, is the distance between the camera and the target, , is the flight height; is the field angle, which is the angle between the half image and the focal length. The formula for calculating the field angle is: wherein is the pixel size, is the image size; In the horizontal direction: is the image shift, the motion of the flying body causes the target point the image point on the image plane moves to the distance on the image plane is the image shift to be solved ; Due to , , In the vertical direction: is the image shift, the motion of the flying body causes the target point the image point on the image plane moves to the distance on the image plane is the image shift to be solved ; Therefore so wherein is ; The key point photographs collected by each different aircraft in the group of cooperative search form the same named image point observation of the same space point in different images.

2. The ship precise positioning method based on three-dimensional depth feature extraction according to claim 1, characterized in that, In the step S3: Convert all received photographic pose data to the body coordinate system, use the body coordinate system of the aircraft as the reference coordinate system for joint solving, and use the key point photograph coordinates identified by itself and the key point photograph coordinates received from other aircrafts as the image point observation data required for joint solving; The photographic pose of the own aerial vehicle and the photographic pose of the other aerial vehicles are taken as control conditions, and all the three-dimensional space coordinates of the key points in the aerial vehicle body coordinate system and three-dimensional feature information are solved by joint block adjustment technology to identify the target position and calculate the direction vector of the attack.

3. The ship precise positioning method based on three-dimensional depth feature extraction according to claim 1, characterized in that, In the step S4: After determining the type of the ship by using the three-dimensional information of the identified key points on the image, the three-dimensional model of the ship is placed into the reference coordinate system of the aerial vehicle according to the space coordinates of the key points solved by the block adjustment, and any specified position on the ship is positioned.

4. A ship precise positioning system based on three-dimensional depth feature extraction, characterized in that, Comprise: Module M1: based on the principle of optical stereo imaging, combined with photographic pose data, obtain multi-view images imaged at different angles and different distances; Module M2: identify the region of interest frame in the obtained image, frame the ship target in the image, and identify the key points in the region of interest; Module M3: identify the key points containing semantics, and pair the key points identified in the images taken at different angles according to semantic labels; Module M4: classify and identify the target by using the three-dimensional space coordinate information of the key points, identify the ship model, and virtually place the real model in the database into the body coordinate system according to the position of the key points in space; In the module M1: When the target meets multiple aerial vehicle trajectory conditions, multiple aerial vehicles or multiple aerial vehicle trajectory mixed cooperation are used according to tactical needs; When multiple aerial vehicles cooperate in reconnaissance, they communicate with each other, use the information obtained by multiple aerial vehicles to jointly solve the three-dimensional information of the target for identification and key point positioning; communication adopts a broadcast mode, each aerial vehicle broadcasts the information obtained by itself to other aerial vehicles, and receives the information sent by other aerial vehicles, each aerial vehicle obtains complete data, and independently solves after receiving all the data; Each aerial vehicle takes a total of three photographs at the end of positioning, the first photograph is taken when the aerial vehicle is farthest from the target, the field of view is largest, and the target recognition algorithm can determine the search target from multiple targets; After the first photograph, the flight control system adjusts the flight direction according to the positioning result, the target appears in the center area of the image during subsequent photography, the search range of the target region of interest frame is reduced, and the data processing time is shortened; at the same time, when the accuracy of the region of interest frame range is greater than the preset standard, the actual camera imaging switches to the interested region imaging mode, and only images within the interested region range; According to the pre-identified and calibrated feature points of the ship models in the database, each aerial vehicle takes a total of three photographs at the end of search, and after each photograph, the following steps are performed: Identify the region of interest frame in the obtained image, frame the ship target in the image, and identify the key points containing semantics in the region of interest; The key points identified in the images taken at different angles contain three-dimensional features, and are paired according to semantic labels; Send the photograph coordinates of the identified key points and the photographic pose parameters recorded by the navigation device on the missile during photography to all other aerial vehicles, and receive the data sent by other aerial vehicles at the same time, the photographic pose includes exposure position and photographic pose; Pose data solving: the lens center of the point camera, A point is a target point in the scene space that, during the exposure time , moves relatively to the point: wherein is the angle between the imaging principal axis and the plumb line, aircraft speed, is the flight time; is the focal length of the camera, is the distance between the camera and the target, , is the altitude; is the field angle, which is the angle between the half image and the focal length. The formula for calculating the field angle is: wherein is the pixel size, is the image size; In the horizontal direction: is the image shift, the motion of the flying body causes the target point in the image plane moves to , the distance in the image plane is the image shift required to be solved ; Due to , , In the vertical direction: is the image shift, the motion of the flying body causes the target point the image point on the image plane moves to the distance on the image plane is the image shift to be solved ; Due to therefore wherein is The different aircrafts in this group of cooperative search take the key point photos collected each time to observe the homonymy image points of the same space point on different images.

5. The ship precise positioning system based on three-dimensional deep feature extraction of claim 4, characterized in that, In the module M3: Convert all the received photographic position data to the body coordinate system, take the body coordinate system of the aircraft as the reference coordinate system for joint solution, and take the key point photo coordinates recognized by itself and the key point photo coordinates received from other aircrafts as the image point observation data required for joint solution; Take the photographic position of the aircraft itself and the photographic position of other aircrafts as the control conditions, solve the three-dimensional space coordinates of all key points in the body coordinate system of the aircraft and the three-dimensional feature information through joint block adjustment technology, recognize the target position and calculate the direction vector of the strike.

6. The ship precise positioning system based on three-dimensional deep feature extraction of claim 4, characterized in that, In the module M4: After determining the type of the ship using the three-dimensional information of the key points recognized on the image, the three-dimensional model of the ship is placed into the reference coordinate system of the aircraft according to the key point space coordinates solved by block adjustment, and any specified position on the ship is positioned.

Images