Feature extraction network model training method, image processing method and device

Abstract

The present application discloses a method for training a feature extraction network model, an image processing method and a device thereof, and relates to the field of neural network technology. The method for training the feature extraction network model includes: obtaining a training data set, the training data set includes multiple image frames; based on the proximity relationship between the multiple image frames and the multiple image frames, weakly supervised training of the network model to be trained to obtain a feature extraction network model, wherein the feature extraction network model is used to determine the feature point data corresponding to the image set to be processed based on the image set to be processed. The feature extraction network model obtained by training based on the present application can not only ensure the number of extracted feature points, but also improve the robustness of the extracted feature points. In particular, for images with repeated textures, the feature extraction network model obtained by training in the present application can fully take into account the number and robustness of feature points.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a training method for a feature extraction network model, an image processing method and a device thereof. Background Art

[0002] As we all know, feature point extraction methods for images are widely used in many image processing scenarios, such as farmland mapping and farmland reconstruction.

[0003] However, in general, the robustness of feature points extracted based on existing feature point extraction methods is poor, especially for images with repeated textures. Relying on feature points with poor robustness, it is difficult to guarantee the number of subsequent feature point matching pairs. Without a sufficient number of feature point matching pairs, it is difficult to establish the data association relationship between image frames, and thus it is difficult to ensure the smooth progress of subsequent image processing operations. Summary of the invention

[0004] In view of this, the present application provides a training method for a feature extraction network model, an image processing method and a device thereof to solve the problem that the number of extracted feature points is too small and the robustness is poor.

[0005] In a first aspect, the present application provides a method for training a feature extraction network model, which includes: obtaining a training data set, the training data set including multiple image frames; based on the proximity relationship between the multiple image frames and the multiple image frames, weakly supervising the network model to be trained to obtain a feature extraction network model, wherein the feature extraction network model is used to determine feature point data corresponding to the image set to be processed based on the image set to be processed.

[0006] In a second aspect, the present application provides a method for determining the heading of an aircraft, which includes: obtaining a set of remote sensing images taken by the aircraft during flight; determining feature point data corresponding to the remote sensing image set based on a feature extraction network model, wherein the feature extraction network model is trained based on the method described in the first aspect above; generating feature point matching pair data corresponding to the remote sensing image set based on the feature point data corresponding to the remote sensing image set; and determining the heading data of the aircraft based on the feature point matching pair data corresponding to the remote sensing image set.

[0007] In a third aspect, the present application provides a training device for a feature extraction network model, which includes: an acquisition module for acquiring a training data set, wherein the training data set includes multiple image frames; a training module for weakly supervising the training of the network model to be trained based on the proximity relationship between the multiple image frames and the multiple image frames to obtain a feature extraction network model, wherein the feature extraction network model is used to determine feature point data corresponding to the image set to be processed based on the image set to be processed.

[0008] In a fourth aspect, the present application provides a device for determining a heading of an aircraft, which includes: an acquisition module for acquiring a set of remote sensing images taken by the aircraft during flight; a matching pair determination module for determining feature point data corresponding to the remote sensing image set based on a feature extraction network model, wherein the feature extraction network model is trained based on the method described in the first aspect above; and based on the feature point data corresponding to the remote sensing image set, feature point matching pair data corresponding to the remote sensing image set is generated; the heading determination module is used to determine the heading data of the aircraft based on the feature point matching pair data corresponding to the remote sensing image set.

[0009] In a fifth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed by a processor of an electronic device, the electronic device can execute the method mentioned in the first or second aspect above.

[0010] In a sixth aspect, the present application provides an electronic device, comprising: a processor; a memory for storing computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method mentioned in the first or second aspect above.

[0011] The training method of the feature extraction network model provided in the embodiment of the present application generates a training data set through image sequence data collected by each of M sorties of aircraft, and then based on the training data set, a weakly supervised training method of the network model to be trained is used to train the feature extraction network model. The feature extraction network model trained based on the embodiment of the present application can not only ensure the number of extracted feature points, but also improve the robustness of the extracted feature points, thereby providing a prerequisite for ensuring the number of subsequent feature point matching pairs. In particular, for images with repetitive textures, the feature extraction network model trained in the embodiment of the present application can fully take into account the number and robustness of feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Shown is a schematic diagram of an application scenario of the method provided in one embodiment of the present application.

[0013] Figure 2The figure is a flow chart of a method for training a feature extraction network model provided in one embodiment of the present application.

[0014] Figure 3 The figure is a flow chart of weakly supervising the training of a network model to be trained based on a training data set to obtain a feature extraction network model provided by an embodiment of the present application.

[0015] Figure 4 The figure is a schematic diagram of a process for determining a target loss function from a first loss function and a second loss function according to a proximity relationship between multiple image frames provided by an embodiment of the present application.

[0016] Figure 5 The figure is a schematic diagram of the construction principle of the training data set provided in one embodiment of the present application.

[0017] Figure 6 Shown is a schematic diagram of the construction principle of a training data set provided in another embodiment of the present application.

[0018] Figure 7 The figure is a flow chart of a method for training a feature extraction network model provided in one embodiment of the present application.

[0019] Figure 8 FIG. 1 is a flow chart of an image processing method provided in one embodiment of the present application.

[0020] Fig. 9 The figure is a schematic flow chart of a method for determining the heading of an aircraft provided in one embodiment of the present application.

[0021] Fig.10 Shown is a schematic diagram of the structure of a training device for a feature extraction network model provided in one embodiment of the present application.

[0022] Fig.11 FIG. 4 is a schematic diagram of the structure of an image processing device provided in one embodiment of the present application.

[0023] Fig.12 Shown is a schematic structural diagram of a heading determination device for an aircraft provided in one embodiment of the present application.

[0024] Fig.13 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] At present, commonly used feature point extraction methods include the Scale Invariant Feature Transform (SIFT) method. That is, the SIFT feature points of the image to be processed are extracted, and then feature point matching pairs are generated based on the extracted SIFT feature points. However, the extracted SIFT feature points are often not robust enough to ensure that there are enough feature point matching pairs in the future. Especially for images with repeated textures (such as banana forest images, wheat field images, corn field images, etc.), the extraction of feature points is more difficult. Although lowering the threshold can increase the number of SIFT feature points obtained, it still cannot guarantee the robustness of the feature points.

[0027] In order to solve the above problems, the embodiments of the present application provide a training method for a feature extraction network model, an image processing method and a device thereof to solve problems such as poor robustness of the extracted feature points.

[0028] Combine the following Figure 1 Introduce the specific application scenarios of the method mentioned in this application.

[0029] Figure 1 The figure shows a schematic diagram of an application scenario of the method provided by an embodiment of the present application. Figure 1 As shown, the application scenario provided by the embodiment of the present application is a scenario for determining the heading of an aircraft. Specifically, the scene includes a target area 110 with a repetitive texture and an aircraft 120. The aircraft 120 includes an image acquisition device 111 and a processor 112 connected to the image acquisition device 111. The image acquisition device 111 is used to capture a remote sensing image set during the flight of the aircraft 120. The processor 112 is used to determine the feature point matching pair data corresponding to the remote sensing image set, and determine the heading data of the aircraft 120 based on the feature point matching pair data corresponding to the remote sensing image set. Among them, the heading data includes the flight attitude data (also called posture data) of the aircraft (such as a drone). With such a setting, the accuracy of the heading data can be improved by utilizing feature point data with better robustness.

[0030] Optionally, the method provided in the embodiment of the present application is also applicable to another scenario, in which the image acquisition device 111 takes the remote sensing image set and sends the remote sensing image set to a cloud server or an image processing device, which determines the feature point matching pair data corresponding to the remote sensing image set and determines the heading data of the aircraft 120 based on the feature point matching pair data corresponding to the remote sensing image set.

[0031] Exemplarily, a method for determining feature point matching pair data corresponding to a remote sensing image set includes: determining feature point data corresponding to an image set to be processed (i.e., a remote sensing image set) based on a feature extraction network model, and generating feature point matching pair data corresponding to the image set to be processed based on the feature point data corresponding to the image set to be processed.

[0032] Exemplarily, a training method for a feature extraction network model includes: obtaining a training data set, the training data set including multiple image frames; based on the proximity relationship between the multiple image frames and the multiple image frames, weakly supervised training of the network model to be trained to obtain a feature extraction network model, wherein the feature extraction network model is used to determine feature point data corresponding to the image set to be processed based on the image set to be processed.

[0033] Combine the following Figures 2 to 6 The relevant methods provided in this application are described in detail.

[0034] Figure 2 The figure is a flow chart of a method for training a feature extraction network model provided in an embodiment of the present application. For example, the method for training a feature extraction model provided in an embodiment of the present application can be executed in a processor of a hardware device such as an aircraft, or in a related server.

[0035] like Figure 2 As shown, the training method of the feature extraction network model provided in the embodiment of the present application includes the following steps.

[0036] Step S210: acquiring a training data set, wherein the training data set includes a plurality of image frames.

[0037] Exemplarily, the training data set is generated based on the image sequence data collected by each of the M sorties of aircraft. For example, the image sequence data collected by each of the M sorties of aircraft is directly used as the training data set.

[0038] The flight information of the aircraft includes a lot of information such as the flight date, aircraft number and flight times. It can be seen that based on the flight information of the aircraft (i.e. based on the flight of the aircraft), the image sequence data collected by the aircraft in the flight can be determined. Exemplarily, the scene corresponding to the image sequence data is a farmland scene, and the aircraft includes but is not limited to a plant protection drone and a surveying and mapping drone. Furthermore, the image sequence data corresponds to P image frames with repeated textures, and the image frames include farmland scene image frames, and the farmland scene image frames include at least one of banana forest image frames, wheat field image frames and corn field image frames.

[0039] For image sequence data collected by the same aircraft, the similarity between image frames (especially adjacent image frames) may be relatively high, so there are usually more feature point matching pairs. Correspondingly, for image sequence data collected by different aircraft, the similarity between image sequence data may be relatively low. For example, if the two sets of image sequence data are not the same target scene, the similarity between the two sets of image sequence data may be relatively low or even no similarity.

[0040] In some embodiments, M is a positive integer, that is, the number of flights can be one or more. It can be understood that, usually, the image frames involved in the image sequence data of one flight are also hundreds of images. Therefore, even if there is only one flight, the generated training data set is also very large and rich. Further, in order to enable the model to be trained to fully learn the data association relationship between image frames, M is defined as a positive integer greater than 1, that is, the training data set is generated based on the image sequence data collected by multiple aircraft of different flights.

[0041] Step S220, based on the proximity relationship between the multiple image frames and the multiple image frames, weakly supervise the training network model to obtain a feature extraction network model.

[0042] Exemplarily, the feature extraction network model is used to determine feature point data corresponding to the image set to be processed based on the image set to be processed. It can be understood that, in essence, the image set to be processed has the image characteristics of the image frames in the training data set. For example, the image set to be processed includes multiple image frames with repeated textures, and the multiple image frames with repeated textures include farmland scene image frames taken by the same aircraft.

[0043] In some embodiments, the network model to be trained is a pre-established initial network model that needs to be trained. It can be understood that the difference between the network model to be trained and the feature extraction network model lies in the model parameters. Weak supervision training of the network model to be trained, that is, training the network model to be trained based on weak supervision signals, specifically, generating training constraints (i.e., model loss function) based on weak supervision signals, and then training the network model to be trained based on the training constraints to obtain a feature extraction network model. Correspondingly, in the actual application process, the loss function of the model is used to determine whether the model has converged, that is, to determine whether the model has been trained.

[0044] The training method of the feature extraction network model provided in the embodiment of the present application obtains the feature extraction network model by weakly supervising the training of the network model to be trained based on the proximity relationship between multiple image frames and multiple image frames. The feature extraction network model obtained by training based on the embodiment of the present application can not only ensure the number of extracted feature points, but also improve the robustness of the extracted feature points, thereby providing a prerequisite for ensuring the number of subsequent feature point matching pairs. In particular, for images with repetitive textures, the feature extraction network model obtained by training in the embodiment of the present application can fully take into account the number and robustness of feature points.

[0045] Figure 3 The figure shows a flow chart of weakly supervising the training of the network model to be trained based on the training data set and obtaining the feature extraction network model according to an embodiment of the present application. Figure 3 As shown, in an embodiment of the present application, based on the proximity relationship between multiple image frames and multiple image frames, weakly supervised training of the network model to be trained to obtain a feature extraction network model step includes the following steps.

[0046] Step S221, constructing a first loss function and a second loss function corresponding to the network model to be trained.

[0047] Exemplarily, the first loss function includes a response constraint function, and the second loss function includes a response constraint function and a regularization term.

[0048] Step S222, determining a target loss function from the first loss function and the second loss function according to the proximity relationship between the multiple image frames.

[0049] Combine the following Figure 4 An example is given to illustrate a specific implementation method of determining the target loss function from the first loss function and the second loss function according to the proximity relationship between multiple image frames. Figure 4 As shown, in an embodiment of the present application, the step of determining the target loss function from the first loss function and the second loss function according to the proximity relationship between the multiple image frames includes steps S2221 and S2222. Step S2221, when there is no proximity relationship, the first loss function is determined to be the target loss function. Step S2222, when there is a proximity relationship, the second loss function is determined to be the target loss function.

[0050] Step S223, based on the target loss function and multiple frame images, weakly supervise the network model to be trained to obtain a feature extraction network model.

[0051] The embodiment of the present application can make full use of the proximity relationship between multiple image frames to determine the target loss function. Since the target loss function is determined based on the proximity relationship between multiple image frames, the embodiment of the present application can make the feature point data output by the trained model more accurate.

[0052] Further, in some embodiments, the neighbor relationship includes a positive neighbor relationship and a secondary neighbor relationship. The target loss function is determined from the first loss function and the second loss function according to the neighbor relationship between the multiple frames of image data, including: when there is a neighbor relationship, the second loss function corresponding to the positive neighbor relationship and the second loss function corresponding to the secondary neighbor relationship are obtained; based on the second loss function corresponding to the positive neighbor relationship and the second loss function corresponding to the secondary neighbor relationship, the target loss function is jointly determined. In the actual flight process of the aircraft, if the K moment corresponds to a position, then the position at the K+1 moment may be estimated using the model, and there may be a certain error. If the aircraft keeps flying, the error will continue to accumulate, resulting in an increasing error. In this case, if the neighbor relationship, especially the secondary neighbor relationship, is used, the purpose of using the information of the previous frame or the previous frames to estimate the current frame can be achieved, thereby achieving joint optimization. It can be seen that the addition of the secondary neighbor relationship can further improve the accuracy of the subsequently reconstructed map.

[0053] Combine the following Figure 5 and Figure 6 The meanings of adjacent image frames (also called directly adjacent image frames) and second adjacent image frames are explained with examples.

[0054] Figure 5 FIG. 1 is a schematic diagram showing the construction principle of a training data set provided in an embodiment of the present application. Figure 5 As shown, in the embodiment of the present application, there are two image sequences of two flights, namely flight 1 and flight 2. The image sequence of flight 1 includes three image frames, namely image frame 31, image frame 32 and image frame 33. The image sequence of flight 2 includes three image frames, namely image frame 34, image frame 35 and image frame 36.

[0055] In the same flight, every two adjacent image frames can be regarded as directly adjacent image frames. Specifically, in flight 1, image frame 31 and image frame 32 can be regarded as directly adjacent image frames, and image frame 32 and image frame 33 can be regarded as directly adjacent image frames. Similarly, in flight 2, image frame 34 and image frame 35 can be regarded as directly adjacent image frames, and image frame 35 and image frame 36 can be regarded as directly adjacent image frames. In the same flight, every two image frames separated by one frame can be regarded as sub-adjacent image frames. Specifically, in flight 1, image frame 31 and image frame 33 can be regarded as sub-adjacent image frames. Similarly, in flight 2, image frame 34 and image frame 36 can be regarded as sub-adjacent image frames. In addition, since flight 1 and flight 2 are two different flights, there is no adjacent relationship between the image frames.

[0056] Figure 6 FIG. 2 is a schematic diagram showing the construction principle of a training data set provided by another embodiment of the present application. Figure 6As shown, in the embodiment of the present application, there are two groups of image sequences of different channels of one flight, namely the Sth channel of flight 1 and the S+1th channel of flight 1. The Sth channel and the S+1th channel are adjacent channels (which can be regarded as a pair of round-trip channels), and S can be a positive integer. Specifically, the image sequence of the Sth channel includes three image frames, namely image frame 41, image frame 42, and image frame 43. The image sequence of the S+1th channel includes three image frames, namely image frame 44, image frame 45, and image frame 46.

[0057] In the same channel, every two adjacent image frames can be regarded as directly adjacent image frames. Specifically, in the Sth channel, image frame 41 and image frame 42 can be regarded as directly adjacent image frames, and image frame 42 and image frame 43 can be regarded as directly adjacent image frames. Similarly, in the S+1th channel, image frame 44 and image frame 45 can be regarded as directly adjacent image frames, and image frame 45 and image frame 46 can be regarded as directly adjacent image frames. In the same channel, every two image frames separated by one frame can be regarded as sub-adjacent image frames. Specifically, in the Sth channel, image frame 41 and image frame 43 can be regarded as sub-adjacent image frames. Similarly, in the S+1th channel, image frame 44 and image frame 46 can be regarded as sub-adjacent image frames. In addition, since the Sth channel and the S+1th channel are adjacent channels of the same flight, the image frames can also be regarded as sub-adjacent image frames with a lateral association relationship. Specifically, image frame 41 and image frame 44 may be considered as next adjacent image frames, image frame 42 and image frame 45 may be considered as next adjacent image frames, and image frame 43 and image frame 46 may be considered as next adjacent image frames.

[0058] It should be noted that Figure 5 and Figure 6 This is just an example, and does not limit the meaning of the adjacent image frame and the next adjacent image frame. For example, for the same channel, every two image frames separated by two frames can also be regarded as next adjacent image frames. In addition, the number of image frames included in each channel is not limited to three. Furthermore, the S+1 channel can also be replaced by the S-1 channel, where S is a positive integer greater than 1.

[0059] In summary, in some embodiments, the neighboring relationship includes a positive neighboring relationship and a secondary neighboring relationship. The image frames with a positive neighboring relationship are positive neighboring image frames, and the image frames with a secondary neighboring relationship are secondary neighboring image frames. Positive neighboring image frames include image sequence data collected on the same route for the same flight of aircraft, and every two adjacent image frames in the image sequence data. Secondary adjacent image frames include image sequence data collected on the same route for the same flight of aircraft, and every two image frames separated by a preset number of frames in the image sequence data. And / or, secondary adjacent image frames include two sets of image sequence data collected on adjacent routes for the same flight of aircraft, and every two lateral adjacent image frames in the two sets of image sequence data.

[0060] In some embodiments, the response constraint function is determined based on the number of feature points of the image frames to be matched in the multiple image frames, the minimum distance and the second minimum distance of the feature point matching pairs between the image frames to be matched; the regularization term is determined based on the feature point distance matrix of the image frames to be matched, and the number of the image frames to be matched is greater than or equal to 2. This setting is to further improve the robustness of the determined feature point data.

[0061] The following is an example of two image frames to be matched.

[0062] Specifically, the following is further explained in conjunction with the following formula (1).

[0063]

[0064] In formula (1), loss weakly represents the target loss function, D represents the feature point distance matrix of each two image frames, y represents the weak supervision signal, when y = 0, it represents that the image frames to be matched (i.e., each two image frames) do not have a neighboring relationship, when y = 1, it represents that the image frames to be matched (i.e., each two image frames) have a neighboring relationship. N represents the number of feature points of each two image frames, d1 represents the minimum distance between the feature point matching pairs of each two image frames, and d2 represents the second minimum distance between the feature point matching pairs of each two image frames. Characterizes the regularization term, n m Characterizes the number of matrix elements in the feature point distance matrix where the matrix elements are minimum values ​​in the matrix rows and matrix columns where the matrix elements are located. The feature point distance matrix between every two image frames refers to the distance matrix formed by all feature points in every two image frames. For example, if the first image frame includes two feature points and the second image frame includes two feature points, then the feature point distance matrix between the two image frames is a matrix with two rows and two columns.

[0065] It should be noted that the distance mentioned above refers to the Euclidean distance. It can be understood that the smaller the distance, the higher the matching accuracy. The meaning expressed by formula (1) includes: for each two image frames, if y = 0, that is, there is no neighbor relationship, then the expected ratio can be as close as possible; if y = 1, that is, there is a neighbor relationship or a sub-neighbor relationship, the expected Expectation Approaching This means that it is more recognizable. In addition, it is expected Expectation Approaching 0, that is, the expected n m As big as possible.

[0066] The embodiment of the present application utilizes the above-mentioned loss function to greatly improve the robustness and quantity of feature points extracted by the final feature extraction network model.

[0067] In some embodiments, the network model to be trained includes a feature extractor (also called a feature extraction network) and a descriptor (also called a description network). Correspondingly, for every two image frames in the training data set (i.e., image frames to be matched), the feature extractor and the descriptor both perform the following steps.

[0068] The feature extractor is used to process every two image frames respectively, obtain the score map data, direction map data and scale map data corresponding to each two image frames, and then obtain the key point data corresponding to each two image frames. Furthermore, in order to ensure the training effect, for every two image frames, the score map data, direction map data and scale map data corresponding to the first image frame and the score map data, direction map data and scale map data corresponding to the second image frame should be as similar as possible. For example, the feature extractor includes a receptive field network (RF-Net). The descriptor is used to describe the key point data corresponding to every two image frames and generate feature point data including key point description information. For example, HardNet is used as a descriptor to construct a hard loss function to extract the local image feature information corresponding to the key point.

[0069] It should be noted that the feature extractor and descriptor are not limited to the RF-Net and HardNet mentioned above. As long as they are end-to-end feature extractors and descriptors, they can be applied to the embodiments of the present application.

[0070] Figure 7 FIG. 1 is a flow chart of a method for training a feature extraction network model provided in an embodiment of the present application. Figure 7 As shown, in an embodiment of the present application, before weakly supervising the training of the network model to be trained based on the proximity relationship between multiple image frames and multiple image frames, the following steps are also included.

[0071] Step S211, for image frames to be matched among the plurality of image frames, determining the degree of overlap between at least two image frames, wherein the number of image frames to be matched is greater than or equal to 2.

[0072] In some embodiments, determining the degree of overlap between at least two image frames includes: determining identification information corresponding to each image frame in the image frames to be matched, wherein the identification information includes at least one of image position information, image timestamp information, and image frame number information; and determining the degree of overlap between the image frames to be matched based on the identification information corresponding to each image frame in the image frames to be matched.

[0073] Step S212: if the overlap between the image frames to be matched is greater than a preset overlap threshold, it is determined that there is a proximity relationship between the image frames to be matched.

[0074] Step S213: if the overlap between the image frames to be matched is less than or equal to a preset overlap threshold, it is determined that there is no adjacent relationship between the image frames to be matched.

[0075] The embodiment of the present application can achieve the purpose of determining the neighbor relationship between image frames by using the overlap between image frames. Compared with the prior art, the range of the determined neighbor relationship is wider, which is conducive to further improving the robustness of the trained model.

[0076] Figure 8 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present application. Figure 8 As shown, the image processing method provided in the embodiment of the present application includes the following steps.

[0077] Step S510, determining feature point data corresponding to the image set to be processed based on the feature extraction network model.

[0078] Exemplarily, the feature extraction network model mentioned in step S510 is trained based on the training method of the feature extraction network model mentioned in any of the above embodiments.

[0079] Step S520: generating feature point matching pair data corresponding to the image set to be processed based on the feature point data corresponding to the image set to be processed.

[0080] In some embodiments, the image set to be processed includes a plurality of image frames with repeated textures, and the plurality of image frames with repeated textures include farmland scene image frames taken by the same aircraft. The farmland scene image frames include at least one of banana forest image frames, wheat field image frames, and corn field image frames.

[0081] The image processing method provided in the embodiment of the present application can utilize the feature extraction network model to obtain a sufficient number of feature point data with good robustness, and then generate a sufficient number of feature point matching pairs with good matching degree based on the obtained feature point data, and then establish a good inter-frame data association relationship, thereby providing a favorable data basis for subsequent three-dimensional scene reconstruction and / or aircraft heading determination.

[0082] In some embodiments, based on the feature point data corresponding to the image set to be processed, generating the feature point matching pair data corresponding to the image set to be processed includes: generating the feature point matching pair data based on the feature point data corresponding to the image set to be processed and the nearest neighbor ratio method. That is, the embodiment of the present application uses the nearest neighbor ratio method to screen the feature point matching pairs, thereby improving the accuracy of the feature point matching pairs.

[0083] Fig. 9 FIG. 1 is a flow chart of a method for determining the heading of an aircraft provided in an embodiment of the present application. Fig. 9 As shown, the method for determining the heading of an aircraft provided in an embodiment of the present application includes the following steps.

[0084] Step S610, obtaining a set of remote sensing images taken by the aircraft during flight.

[0085] Exemplarily, the remote sensing image set refers to a sequence of images taken by the aircraft during flight. It can be understood that the remote sensing image set mentioned in step S610 can be regarded as the to-be-processed image set mentioned in the above embodiment.

[0086] Step S620: determining feature point matching pair data corresponding to the remote sensing image set.

[0087] Exemplarily, the feature point matching pair data corresponding to the remote sensing image set mentioned in step S620 is obtained based on the image processing method mentioned in any of the above embodiments. That is, based on the feature extraction network model, the feature point data corresponding to the remote sensing image set is determined, and then based on the feature point data corresponding to the remote sensing image set, the feature point matching pair data corresponding to the remote sensing image set is generated. Among them, the feature extraction network model is trained based on the method described in any of the above embodiments.

[0088] Step S630, determining the heading data of the aircraft based on the feature point matching pair data corresponding to the remote sensing image set.

[0089] Since the image sequence taken by the aircraft during flight is continuous, the difference between every two adjacent images can be determined based on the feature point matching pair data between the two adjacent images, and then the heading data of the aircraft can be determined based on the difference between every two adjacent images in the image sequence. Compared with the prior art, the embodiment of the present application can further improve the accuracy of the determined heading data of the aircraft.

[0090] Combination of the above Figures 2 to 9 , describes the method embodiment of the present application in detail, and the following is combined with Figures 10 to 13 , describes the device embodiment of the present application in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0091] Fig.10 FIG. 1 is a schematic diagram of the structure of a training device for a feature extraction network model provided in an embodiment of the present application. Fig.10As shown, the training device of the feature extraction network model provided in the embodiment of the present application includes an acquisition module 710 and a training module 720. The acquisition module 710 is used to acquire a training data set, and the training data set includes multiple image frames. The training module 720 is used to weakly supervise the training of the network model to be trained based on the proximity relationship between the multiple image frames and the multiple image frames to obtain the feature extraction network model, wherein the feature extraction network model is used to determine the feature point data corresponding to the image set to be processed based on the image set to be processed.

[0092] Fig.11 FIG. 1 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present application. Fig.11 As shown, the image processing device provided in the embodiment of the present application includes a determination module 810 and a matching pair generation module 820. The determination module 810 is used to determine the feature point data corresponding to the image set to be processed based on the feature extraction network model. The feature extraction network model is trained based on the training method of the feature extraction network model mentioned in any of the above embodiments. The matching pair generation module 820 is used to generate feature point matching pair data corresponding to the image set to be processed based on the feature point data corresponding to the image set to be processed.

[0093] In some embodiments, the matching pair generation module 820 is further used to generate feature point matching pair data based on feature point data corresponding to the image set to be processed and the nearest neighbor ratio method.

[0094] Fig.12 FIG. 1 is a schematic diagram of the structure of a device for determining the heading of an aircraft provided by an embodiment of the present application. Fig.12 As shown, the heading determination device for an aircraft provided in an embodiment of the present application includes an acquisition module 910, a matching pair determination module 920 and a heading determination module 930. The acquisition module 910 is used to acquire a set of remote sensing images taken by the aircraft during flight. The matching pair determination module 920 is used to determine the feature point data corresponding to the remote sensing image set based on a feature extraction network model; and based on the feature point data corresponding to the remote sensing image set, generate feature point matching pair data corresponding to the remote sensing image set. Among them, the feature extraction network model is trained based on the training method of the feature extraction network model described in any of the above embodiments. The heading determination module 930 is used to determine the heading data of the aircraft based on the feature point matching pair data corresponding to the remote sensing image set.

[0095] In some embodiments, an aircraft is also provided, which includes the aircraft heading determination device mentioned in any of the above embodiments.

[0096] Fig.13 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.13As shown, the electronic device 1000 provided in the embodiment of the present application includes: a processor 1010; and a memory 1020, in which computer program instructions are stored. When the computer program instructions are executed by the processor 1010, the processor 1010 executes the method mentioned in the above embodiment.

[0097] The processor 1010 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0098] The memory 1020 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1010 may run the program instructions to implement the functions of the methods mentioned in the above-mentioned embodiments of the present application described above. Various contents such as feature extraction network models may also be stored in the computer-readable storage medium.

[0099] In one example, the electronic device 1000 may further include: an input device 1030 and an output device 1040 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0100] The input device 1030 may include, for example, a keyboard, a mouse, a microphone, and the like.

[0101] The output device 1040 can output various information to the outside, including feature point data, etc. The output device 1040 can include, for example, a display, a communication network, a speaker and a remote output device connected thereto, etc.

[0102] Of course, to simplify, Fig.13 Only some of the components related to the present application in the electronic device 1000 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 1000 may also include any other appropriate components.

[0103] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0104] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0105] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0106] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0107] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0108] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0109] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0110] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0111] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for training a feature extraction network model, characterized in that: include: Acquire a training data set, wherein the training data set includes a plurality of image frames; Based on the proximity relationship between the multiple image frames and the multiple image frames, weakly supervise the training of the network model to be trained to obtain a feature extraction network model, wherein the feature extraction network model is used to determine feature point data corresponding to the image set to be processed based on the image set to be processed; The weakly supervised training of the network model to be trained based on the proximity relationship between the multiple image frames and the multiple image frames to obtain the feature extraction network model includes: Constructing a first loss function and a second loss function corresponding to the network model to be trained; When there is no neighbor relationship, determining the first loss function as the target loss function; When there is a neighboring relationship, determining the second loss function as the target loss function; Wherein, the first loss function includes a response constraint function, and the second loss function includes the response constraint function and a regularization term; Based on the target loss function and the multiple image frames, weakly supervise the network model to be trained to obtain the feature extraction network model; Wherein, the objective loss function includes: Wherein, D represents the feature point distance matrix of the image frames to be matched in the multiple image frames, y represents the weak supervisory signal between the image frames to be matched in the training data set, when y=0, it represents that the image frames to be matched do not have a neighboring relationship, when y=1, it represents that the image frames to be matched have a neighboring relationship; N represents the number of feature points of every two image frames, d1 represents the minimum distance between the feature point matching pairs of the image frames to be matched, d2 represents the second smallest distance between the feature point matching pairs of the image frames to be matched, and n m The number of matrix elements in the matrix rows and matrix columns where the matrix elements are located in the feature point distance matrix of the image frame to be matched, where the matrix elements are all minimum values; The neighboring relationship includes a positive neighboring relationship and a secondary neighboring relationship. The image frame having the positive neighboring relationship is a positive neighboring image frame, and the image frame having the secondary neighboring relationship is a secondary neighboring image frame. The adjacent image frames include image sequence data collected from aircraft of the same sortie on the same route, and every two adjacent image frames in the image sequence data; The secondary adjacent image frames include image sequence data collected from the same flight of aircraft on the same route, and every two image frames in the image sequence data are separated by a preset number of frames; and / or, The secondary adjacent image frames include two sets of image sequence data collected from aircraft of the same sortie on adjacent routes, and each two laterally adjacent image frames in the two sets of image sequence data.

2. The method for training a feature extraction network model according to claim 1, characterized in that: Before weakly supervising the training of the network model to be trained based on the proximity relationship between the multiple image frames and the multiple image frames, the method further includes: For the image frames to be matched among the multiple image frames, determining the overlap between the image frames to be matched; the number of the image frames to be matched is greater than or equal to 2; If the overlap between the image frames to be matched is greater than a preset overlap threshold, determining that the proximity relationship exists between the image frames to be matched; If the overlap between the image frames to be matched is less than or equal to the preset overlap threshold, it is determined that the neighboring relationship does not exist between the image frames to be matched.

3. The training method of the feature extraction network model according to claim 2, characterized in that: The determining the overlap between the image frames to be matched comprises: Determine identification information corresponding to each image frame in the image frames to be matched, wherein the identification information includes at least one of image position information, image timestamp information, and image frame number information; Based on identification information corresponding to each image frame in the image frames to be matched, the overlap degree between the image frames to be matched is determined.

4. The method for training a feature extraction network model according to any one of claims 1 to 3, characterized in that: The network model to be trained includes a feature extractor and a descriptor. For the image frames to be matched in the training data set, The feature extractor is used to process the image frames to be matched respectively to obtain score map data, direction map data and scale map data corresponding to each image frame in the image frames to be matched, and then obtain key point data corresponding to the image frames to be matched; The descriptor is used to describe the key point data corresponding to the image frame to be matched, and generate feature point data including key point description information.

5. The method for training a feature extraction network model according to any one of claims 1 to 3, characterized in that: The multiple image frames include image frames with repeated textures, and the multiple image frames include farmland scene image frames taken by the same flight of aircraft.

6. A method for determining the heading of an aircraft, characterized in that: include: Obtain a set of remote sensing images taken by the aircraft during flight; Determine feature point data corresponding to the remote sensing image set based on a feature extraction network model, wherein the feature extraction network model is trained based on the method described in any one of claims 1 to 5 above; Based on the feature point data corresponding to the remote sensing image set, generating feature point matching pair data corresponding to the remote sensing image set; The heading data of the aircraft is determined based on the feature point matching pair data corresponding to the remote sensing image set.

7. A training device for a feature extraction network model, characterized in that: include: An acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of image frames; A training module, used for weakly supervising the training of the network model to be trained based on the proximity relationship between the multiple image frames and the multiple image frames to obtain a feature extraction network model, wherein the feature extraction network model is used to determine feature point data corresponding to the image set to be processed based on the image set to be processed; The training module is also used to construct a first loss function and a second loss function corresponding to the network model to be trained; when there is no neighbor relationship, the first loss function is determined to be the target loss function; when there is a neighbor relationship, the second loss function is determined to be the target loss function; wherein the first loss function includes a response constraint function, and the second loss function includes the response constraint function and a regularization term; based on the target loss function and the multiple image frames, the network model to be trained is weakly supervised to obtain the feature extraction network model; Wherein, the objective loss function includes: Wherein, D represents the feature point distance matrix of the image frames to be matched in the multiple image frames, y represents the weak supervisory signal between the image frames to be matched in the training data set, when y=0, it represents that the image frames to be matched do not have a neighboring relationship, when y=1, it represents that the image frames to be matched have a neighboring relationship; N represents the number of feature points of every two image frames, d1 represents the minimum distance between the feature point matching pairs of the image frames to be matched, d2 represents the second smallest distance between the feature point matching pairs of the image frames to be matched, and n m The number of matrix elements in the matrix rows and matrix columns where the matrix elements are located in the feature point distance matrix of the image frame to be matched, where the matrix elements are all minimum values; The neighboring relationship includes a positive neighboring relationship and a secondary neighboring relationship. The image frame having the positive neighboring relationship is a positive neighboring image frame, and the image frame having the secondary neighboring relationship is a secondary neighboring image frame. The adjacent image frames include image sequence data collected from aircraft of the same sortie on the same route, and every two adjacent image frames in the image sequence data; The secondary adjacent image frames include image sequence data collected from the same flight of aircraft on the same route, and every two image frames in the image sequence data are separated by a preset number of frames; and / or, The secondary adjacent image frames include two sets of image sequence data collected from aircraft of the same sortie on adjacent routes, and each two laterally adjacent image frames in the two sets of image sequence data.

8. A device for determining the heading of an aircraft, characterized in that: include: An acquisition module is used to acquire a set of remote sensing images taken by the aircraft during flight; A matching pair determination module, for determining feature point data corresponding to a remote sensing image set based on a feature extraction network model, wherein the feature extraction network model is trained based on the method described in any one of claims 1 to 5 above; and generating feature point matching pair data corresponding to the remote sensing image set based on the feature point data corresponding to the remote sensing image set; The heading determination module is used to determine the heading data of the aircraft based on the feature point matching pair data corresponding to the remote sensing image set.

9. A computer-readable storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any one of claims 1 to 5 or the method for determining the heading of an aircraft described in claim 6.

10. An electronic device, characterized in that: include: processor; memory for storing computer executable instructions; The processor is used to execute the computer executable instructions to implement the method described in any one of claims 1 to 5 or the method for determining the heading of an aircraft described in claim 6.

Images