Image information extraction method, model training method, device, equipment and medium

Abstract

The present disclosure provides an image information extraction method, a model training method, an apparatus, a device and a medium. The image information extraction method is executed based on a trained image information extraction model, including: inputting a target image into a feature extraction network in a feature extraction module to obtain an original feature map output by the feature extraction network; inputting the feature map extracted by the feature extraction module into a feature point detection network and a descriptor generation network respectively to obtain a feature point detection result vector and a local descriptor vector, wherein the feature map extracted by the feature extraction module is the original feature map; obtaining a feature point detection result image based on the feature point detection result vector; obtaining a local descriptor result based on the local descriptor vector; when training the image information extraction model, using an upsampling module in the feature extraction module to upsample the original feature map of the sample output by the feature extraction network to obtain a sample intermediate feature map, and using the sample intermediate feature map as the sample feature map extracted by the feature extraction module.

Description

Technical Field

[0001] The present disclosure generally relates to the field of image technology, and more specifically, to an image information extraction method, a model training method, an apparatus, a device and a medium. Background Art

[0002] In image feature matching tasks, SIFT / ORB and other methods are generally used to extract feature points and generate image descriptors, but these traditional methods perform poorly for images with motion blur or weak textures. The emergence of deep learning has provided some new ideas for image feature matching, but the deep learning algorithms currently proposed have a slow speed for feature point detection and matching reasoning, making it difficult to meet the real-time needs of clients. Summary of the invention

[0003] An exemplary embodiment of the present disclosure is to provide an image information extraction method, a model training method, an apparatus, a device and a medium to at least solve at least one technical problem in the above-mentioned related technologies.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image information extraction method, the image information extraction method being performed based on a trained image information extraction model, the image information extraction model comprising: a feature extraction module, a feature point detection network and a descriptor generation network, wherein the image information extraction method comprises: inputting a target image into the feature extraction network in the feature extraction module to obtain an original feature map output by the feature extraction network; inputting the feature map extracted by the feature extraction module into the feature point detection network and the descriptor generation network respectively to obtain a feature point detection result vector output by the feature point detection network and a local descriptor vector output by the descriptor generation network, wherein the feature extraction module The extracted feature map is the original feature map; based on the feature point detection result vector, a feature point detection result image with a resolution equal to the original resolution of the target image is obtained, wherein the pixel value of each pixel in the feature point detection result image represents the probability that the pixel in the target image is a feature point; based on the local descriptor vector, a local descriptor result with a resolution equal to the original resolution of the target image is obtained; wherein, when training the image information extraction model, the upsampling module in the feature extraction module is used to upsample the original feature map of the sample output by the feature extraction network to obtain an intermediate feature map of the sample, wherein the intermediate feature map of the sample is used as the sample feature map extracted by the feature extraction module.

[0005] Optionally, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample, where N is an even number; when training the image information extraction model, the resolution of the feature point detection result image of the sample image is N times the size of the original resolution of the sample image; when training the image information extraction model, the resolution of the local descriptor result of the sample image is N times the size of the original resolution of the sample image.

[0006] Optionally, the upsampling process is: fixed interpolation upsampling process.

[0007] Optionally, the image information extraction model further includes: a global descriptor acquisition module, wherein the image information extraction method further includes: inputting the original feature map into the global descriptor acquisition module to obtain a global descriptor result of the target image.

[0008] Optionally, the step of inputting the original feature map into the global descriptor acquisition module to obtain a global descriptor result of the target image includes: inputting the original feature map into the global descriptor acquisition module, the global descriptor acquisition module compressing the original feature map into a one-dimensional vector, and performing a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the target image.

[0009] Optionally, the feature extraction network includes multiple convolutional neural networks for downsampling processing, wherein each convolutional neural network for downsampling processing uses separable convolution and has an inverted residual spindle structure.

[0010] Optionally, based on the feature point detection result vector, the step of obtaining a feature point detection result image with a resolution equal to the original resolution of the target image includes: passing the feature point detection result vector through a convolutional neural network and performing multi-channel reorganization to obtain a feature point detection result image with a resolution equal to the original resolution of the target image.

[0011] Optionally, based on the local descriptor vector, the step of obtaining a local descriptor result having a resolution equal to the original resolution of the target image includes: performing interpolation upsampling processing on the local descriptor vector to obtain a local descriptor result having a resolution equal to the original resolution of the target image.

[0012] Optionally, the target image is an original image or an image portion including a specific object cropped from the original image.

[0013] Optionally, when the target image is an image portion including a specific object cropped from an original image, the image information extraction method further includes: inputting the original image into a target detection network to obtain the area where the specific object is located in the original image; cropping the area from the original image as the target image; wherein the target detection network is an SSD network structure, and each convolutional neural network in the target detection network uses separable convolution.

[0014] According to a second aspect of an embodiment of the present disclosure, a training method for an image information extraction model is provided, wherein the image information extraction model includes: a feature extraction module, a feature point detection network, and a descriptor generation network, and the training method includes: obtaining training samples, wherein the training samples include: a first sample image with annotations and a second sample image with annotations, wherein the second sample image is obtained by transforming the first sample image, and the annotations include: the positions of the feature points in the image; for each sample image in the first sample image and the second sample image, inputting the sample image into the feature extraction network in the feature extraction module, and using the upsampling module in the feature extraction module to upsample the original feature map of the sample output by the feature extraction network to obtain an intermediate feature map of the sample; and inputting the sample feature maps extracted by the feature extraction module into the feature extraction network, respectively. A feature point detection network and the descriptor generation network are used to obtain a feature point detection result vector output by the feature point detection network and a local descriptor vector output by the descriptor generation network, wherein the sample feature map extracted by the feature extraction module is the sample intermediate feature map; based on the feature point detection result vector, a feature point detection result image with a resolution N times the original resolution of the sample image is obtained; based on the local descriptor vector, a local descriptor result with a resolution N times the original resolution of the sample image is obtained; based on the feature point detection result images, local descriptor results, and annotations of the first sample image and the second sample image, the image information extraction model is trained; wherein, when performing image information extraction based on the trained image information extraction model, the original feature map is used as the feature map extracted by the feature extraction module; wherein N is an even number.

[0015] Optionally, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample; when image information extraction of the target image is performed based on the trained image information extraction model, the resolution of the feature point detection result image of the target image is the original resolution size of the target image; when image information extraction of the target image is performed based on the trained image information extraction model, the resolution of the local descriptor result of the target image is the original resolution size of the target image.

[0016] Optionally, the image information extraction model further includes: a global descriptor acquisition module, wherein the training method further includes: inputting the sample original feature map of the first sample image into the global descriptor acquisition module to obtain the global descriptor result of the first sample image, and inputting the sample original feature map of the second sample image into the global descriptor acquisition module to obtain the global descriptor result of the second sample image; wherein based on the feature point detection result images, local descriptor results, and annotations of the first sample image and the second sample image, the step of training the image information extraction model includes: based on the local descriptor result of the first sample image and the local descriptor of the second sample image The method comprises the following steps: determining a first loss function based on the result of the feature point detection of the first sample image and the annotation of the first sample image, and determining a second loss function based on the feature point detection result image of the second sample image and the annotation of the second sample image; determining a third loss function based on the global descriptor result of the first sample image and the global descriptor result of the second sample image; determining a total loss function based on the first loss function, the second loss function, and the third loss function; and training the image information extraction model by adjusting the parameters of the feature extraction module, the feature point detection network, the descriptor generation network, and the global descriptor acquisition module according to the total loss function.

[0017] Optionally, the sample original feature map of the first sample image is input into the global descriptor acquisition module to obtain the global descriptor result of the first sample image, and the sample original feature map of the second sample image is input into the global descriptor acquisition module. The step of obtaining the global descriptor result of the second sample image includes: inputting the sample original feature map of the first sample image into the global descriptor acquisition module, the global descriptor acquisition module compressing the sample original feature map of the first sample image into a one-dimensional vector, and performing a linear transformation on the compressed one-dimensional vector to obtain the global descriptor result of the first sample image; inputting the sample original feature map of the second sample image into the global descriptor acquisition module, the global descriptor acquisition module compressing the sample original feature map of the second sample image into a one-dimensional vector, and performing a linear transformation on the compressed one-dimensional vector to obtain the global descriptor result of the second sample image.

[0018] Optionally, the feature extraction network includes multiple convolutional neural networks for downsampling processing, wherein each convolutional neural network for downsampling processing uses separable convolution and has an inverted residual spindle structure.

[0019] Optionally, the upsampling process is: fixed interpolation upsampling process.

[0020] Optionally, based on the feature point detection result vector, the step of obtaining a feature point detection result image with a resolution N times the original resolution of the sample image includes: passing the feature point detection result vector through a convolutional neural network and performing multi-channel reorganization to obtain a feature point detection result image with a resolution N times the original resolution of the sample image.

[0021] Optionally, based on the local descriptor vector, the step of obtaining a local descriptor result with a resolution N times the original resolution of the sample image includes: performing interpolation upsampling processing on the local descriptor vector to obtain a local descriptor result with a resolution N times the original resolution of the sample image.

[0022] Optionally, the first sample image is an original image or an image portion including a specific object cropped from the original image.

[0023] According to a third aspect of an embodiment of the present disclosure, there is provided an image information extraction device, which performs operations based on a trained image information extraction model, wherein the image information extraction model comprises: a feature extraction module, a feature point detection network and a descriptor generation network, wherein the image information extraction device comprises: a feature extraction unit, which is configured to input a target image into the feature extraction network in the feature extraction module, and obtain an original feature map output by the feature extraction network; an information extraction unit, which is configured to input the feature map extracted by the feature extraction module into the feature point detection network and the descriptor generation network, respectively, and obtain a feature point detection result vector output by the feature point detection network and a local descriptor vector output by the descriptor generation network, wherein the feature extraction module extracts The feature map is the original feature map; the feature point acquisition unit is configured to obtain a feature point detection result image with a resolution equal to the original resolution of the target image based on the feature point detection result vector, wherein the pixel value of each pixel in the feature point detection result image represents the probability that the pixel in the target image is a feature point; the local descriptor acquisition unit is configured to obtain a local descriptor result with a resolution equal to the original resolution of the target image based on the local descriptor vector; wherein, when the training device trains the image information extraction model, the upsampling module in the feature extraction module is used to upsample the original feature map of the sample output by the feature extraction network to obtain an intermediate feature map of the sample, wherein the intermediate feature map of the sample is used as the sample feature map extracted by the feature extraction module.

[0024] Optionally, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample, where N is an even number; when training the image information extraction model, the resolution of the feature point detection result image of the sample image is N times the size of the original resolution of the sample image; when training the image information extraction model, the resolution of the local descriptor result of the sample image is N times the size of the original resolution of the sample image.

[0025] Optionally, the upsampling process is: fixed interpolation upsampling process.

[0026] Optionally, the image information extraction model also includes: a global descriptor acquisition module, wherein the image information extraction device also includes: a global descriptor acquisition unit, configured to input the original feature map into the global descriptor acquisition module to obtain a global descriptor result of the target image.

[0027] Optionally, the global descriptor acquisition unit is configured to input the original feature map into the global descriptor acquisition module, the global descriptor acquisition module compresses the original feature map into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the target image.

[0028] Optionally, the feature extraction network includes multiple convolutional neural networks for downsampling processing, wherein each convolutional neural network for downsampling processing uses separable convolution and has an inverted residual spindle structure.

[0029] Optionally, the feature point acquisition unit is configured to pass the feature point detection result vector through a convolutional neural network and perform multi-channel reorganization to obtain a feature point detection result image with a resolution equal to the original resolution of the target image.

[0030] Optionally, the local descriptor acquisition unit is configured to perform interpolation upsampling processing on the local descriptor vector to obtain a local descriptor result with a resolution equal to the original resolution of the target image.

[0031] Optionally, the target image is an original image or an image portion including a specific object cropped from the original image.

[0032] Optionally, when the target image is an image portion including a specific object cropped from an original image, the image information extraction device further includes: a target image acquisition unit, configured to input the original image into a target detection network to obtain an area where the specific object is located in the original image; and crop the area from the original image as the target image; wherein the target detection network is an SSD network structure, and each convolutional neural network in the target detection network uses separable convolution.

[0033] According to a fourth aspect of an embodiment of the present disclosure, there is provided a training device for an image information extraction model, the image information extraction model comprising: a feature extraction module, a feature point detection network, and a descriptor generation network, the training device comprising: a training sample acquisition unit, configured to acquire training samples, wherein the training samples comprise: a first sample image with annotations and a second sample image with annotations, wherein the second sample image is obtained by transforming the first sample image, and the annotations comprise: the positions of feature points in the image; a feature extraction unit, configured to, for each of the first sample image and the second sample image, input the sample image into a feature extraction network in the feature extraction module, and use an upsampling module in the feature extraction module to upsample the original feature map of the sample output by the feature extraction network to obtain an intermediate feature map of the sample; an information extraction unit, configured to input the sample feature maps extracted by the feature extraction module into the feature point detection network and the descriptor generation network, respectively. The method comprises the following steps: a first sample image detection unit and a second sample image detection unit, wherein the feature point detection network and the descriptor generation network are used to obtain a feature point detection result vector output by the feature point detection network and a local descriptor vector output by the descriptor generation network, wherein the sample feature map extracted by the feature extraction module is the sample intermediate feature map; a feature point acquisition unit is configured to obtain a feature point detection result image with a resolution N times the original resolution of the sample image based on the feature point detection result vector; a local descriptor acquisition unit is configured to obtain a local descriptor result with a resolution N times the original resolution of the sample image based on the local descriptor vector; a training unit is configured to train the image information extraction model based on the feature point detection result images, local descriptor results, and annotations of the first sample image and the second sample image; wherein, when the image information extraction device performs image information extraction based on the trained image information extraction model, the original feature map is used as the feature map extracted by the feature extraction module; wherein N is an even number.

[0034] Optionally, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample; when image information extraction of the target image is performed based on the trained image information extraction model, the resolution of the feature point detection result image of the target image is the original resolution size of the target image; when image information extraction of the target image is performed based on the trained image information extraction model, the resolution of the local descriptor result of the target image is the original resolution size of the target image.

[0035] Optionally, the image information extraction model also includes: a global descriptor acquisition module, wherein the training device also includes: a global descriptor acquisition unit, configured to input the sample original feature map of the first sample image into the global descriptor acquisition module to obtain the global descriptor result of the first sample image, and input the sample original feature map of the second sample image into the global descriptor acquisition module to obtain the global descriptor result of the second sample image; wherein the training unit is configured to: determine a first loss function based on the local descriptor result of the first sample image and the local descriptor result of the second sample image; determine a second loss function based on the feature point detection result image of the first sample image and the annotation of the first sample image, and the feature point detection result image of the second sample image and the annotation of the second sample image; determine a third loss function based on the global descriptor result of the first sample image and the global descriptor result of the second sample image; determine a total loss function based on the first loss function, the second loss function, and the third loss function; and train the image information extraction model by adjusting the parameters of the feature extraction module, the feature point detection network, the descriptor generation network, and the global descriptor acquisition module according to the total loss function.

[0036] Optionally, the global descriptor acquisition unit is configured to input the sample original feature map of the first sample image into the global descriptor acquisition module, the global descriptor acquisition module compresses the sample original feature map of the first sample image into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the first sample image; input the sample original feature map of the second sample image into the global descriptor acquisition module, the global descriptor acquisition module compresses the sample original feature map of the second sample image into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the second sample image.

[0037] Optionally, the feature extraction network includes multiple convolutional neural networks for downsampling processing, wherein each convolutional neural network for downsampling processing uses separable convolution and has an inverted residual spindle structure.

[0038] Optionally, the upsampling process is: fixed interpolation upsampling process.

[0039] Optionally, the feature point acquisition unit is configured to pass the feature point detection result vector through a convolutional neural network and perform multi-channel reorganization to obtain a feature point detection result image with a resolution N times the original resolution of the sample image.

[0040] Optionally, the local descriptor acquisition unit is configured to perform interpolation upsampling processing on the local descriptor vector to obtain a local descriptor result with a resolution N times the original resolution of the sample image.

[0041] Optionally, the first sample image is an original image or an image portion including a specific object cropped from the original image.

[0042] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the image information extraction method as described above and / or the image information extraction model training method as described above.

[0043] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image information extraction method as described above and / or the training method of the image information extraction model as described above.

[0044] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the image information extraction method as described above and / or the training method of the image information extraction model as described above.

[0045] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: using differentiated resolution output during the training and testing process greatly reduces the prediction time of the model while ensuring the feature point matching effect, thereby meeting the real-time needs of users.

[0046] In addition, the global descriptor and local descriptor of the image can be output simultaneously, so that the image can be retrieved globally and feature matching of the image can be retrieved.

[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0049] Figure 1 A flowchart showing a method for enhancing image information rights according to an exemplary embodiment of the present disclosure is shown;

[0050] Figure 2 A flowchart showing a method for training an image information extraction model according to an exemplary embodiment of the present disclosure;

[0051] Figure 3 A comparative schematic diagram showing an image information extraction method and an image information extraction model training method according to an exemplary embodiment of the present disclosure;

[0052] Figure 4 A structural block diagram showing an image information extraction device according to an exemplary embodiment of the present disclosure;

[0053] Figure 5 A structural block diagram showing a training device for an image information extraction model according to an exemplary embodiment of the present disclosure;

[0054] Figure 6 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0055] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0057] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0058] Figure 1 A flow chart of an image information extraction method according to an exemplary embodiment of the present disclosure is shown. As an example, the image information extraction method may be executed by a client.

[0059] The image information extraction method is performed based on a trained image information extraction model, which includes: a feature extraction module, a feature point detection network, and a descriptor generation network. As an example, the image information extraction model can be used in combination with Figure 5 The training method described is used for training.

[0060] Reference Figure 1 In step S101, the target image is input into the feature extraction network in the feature extraction module to obtain the original feature map output by the feature extraction network.

[0061] As an example, the feature extraction network may include multiple convolutional neural networks for downsampling processing. As an example, each convolutional neural network for downsampling processing may use separable convolution and have an inverted residual spindle structure, for example, the structure of each convolutional neural network for downsampling processing may be an inverted residual block. In addition, as an example, each convolutional neural network for downsampling processing may use deformable convolution.

[0062] As an example, the feature extraction network may include three convolutional neural networks for downsampling processing, that is, the feature extraction network performs three downsampling processes on the input target image. For example, when the target image is an image of size 640*360, the feature extraction network performs three downsampling processes on it and outputs an image feature map of 16*80*45.

[0063] The feature extraction network as a whole uses separable convolution to reduce the amount of calculation, and uses the Inverted residuals spindle structure to expand the network dimension and increase the network's fitting ability while ensuring that the amount of calculation increases less. A jump connection structure is added between the three layers of the network with different scales to fuse the information at different resolution levels, so that the network can better combine the local information and global information of the image.

[0064] As an example, the target image may be an original image (ie, original image) or an image portion including a specific object cropped from the original image.

[0065] As an example, when the target image is an image portion including a specific object cropped from the original image, the image information extraction method according to an exemplary embodiment of the present disclosure may also include: inputting the original image into a target detection network to obtain the area where the specific object is located in the original image; and cropping the area from the original image as the target image. As an example, the target detection network may be an SSD network structure. As an example, each convolutional neural network in the target detection network may use separable convolution. It should be understood that the target detection network is only used when it is necessary to match a specific object (e.g., an object) in the image. If the entire image is matched directly, there is no need to detect the area where the specific object is located first, and the original image can be directly input into the image information extraction model for feature extraction.

[0066] In step S102, the feature map extracted by the feature extraction module is input into the feature point detection network and the descriptor generation network respectively to obtain the feature point detection result vector output by the feature point detection network and the local descriptor vector output by the descriptor generation network, wherein the feature map extracted by the feature extraction module is the original feature map.

[0067] As an example, the feature point detection network may be an Interest Point Decoder.

[0068] As an example, the descriptor generation network may be a descriptor decoder.

[0069] In step S103, based on the feature point detection result vector, a feature point detection result image having a resolution equal to the original resolution of the target image is obtained, wherein the pixel value of each pixel in the feature point detection result image represents the probability that the pixel in the target image is a feature point.

[0070] As an example, the feature point detection result vector can be passed through a convolutional neural network and reorganized among multiple channels (i.e., pixel shuffle) to obtain a feature point detection result image with a resolution equal to the original resolution of the target image. For example, a feature point detection result vector with a dimension of 64*80*45 can be restored to a 640*360 image by pixel shuffle.

[0071] In step S104, based on the local descriptor vector, a local descriptor result having a resolution equal to the original resolution of the target image is obtained.

[0072] As an example, the local descriptor vector may be subjected to upsampling processing (e.g., interpolation upsampling processing) to obtain a local descriptor result having a resolution equal to the original resolution of the target image. For example, the local descriptor vector may be subjected to two interpolation upsampling processings to obtain a local descriptor result having a resolution equal to the original resolution of the target image. For example, a local descriptor vector of 256*80*46 dimensions may be subjected to two interpolation upsamplings to obtain a local descriptor result of 256*640*360 (256-dimensional local descriptor, corresponding to a resolution of 640*360).

[0073] In the present disclosure, when training the image information extraction model, the upsampling module in the feature extraction module is used to upsample the sample original feature map output by the feature extraction network to obtain a sample intermediate feature map, wherein the sample intermediate feature map is used as the sample feature map extracted by the feature extraction module. As an example, the upsampling process can be: a fixed interpolation upsampling process.

[0074] In other words, in the process of training the image information extraction model, the original feature map output by the feature extraction network needs to be upsampled by the upsampling module before being input into the feature point detection network and the descriptor generation network. That is, compared with the training method of the image information extraction model, the image information extraction method according to the exemplary embodiment of the present disclosure reduces the step of upsampling the original feature map output by the feature extraction network.

[0075] Accordingly, as an example, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample, where N is an even number; when training the image information extraction model, the resolution of the feature point detection result image of the sample image is N times the size of the original resolution of the sample image; when training the image information extraction model, the resolution of the local descriptor result of the sample image is N times the size of the original resolution of the sample image.

[0076] Accordingly, as an example, when the resolution size of the original feature map extracted when training the image information extraction model and executing the image information extraction method is the same (i.e., for the original feature map of the same size extracted), the resolution of the image feature map input to the feature point detection network and the descriptor generation network when training the image information extraction model is: N times the resolution of the image feature map input to the feature point detection network and the descriptor generation network when executing the image information extraction method. In other words, the amount of calculation of the feature point detection network and the descriptor generation network during prediction is reduced by N times relative to that during training.

[0077] Accordingly, as an example, when the resolution of the original feature map extracted when training the image information extraction model and when executing the image information extraction method is the same, the resolution (i.e., size) of the feature point detection result vector output by the feature point detection network when training the image information extraction model is: N times the resolution of the feature point detection result vector output by the feature point detection network when executing the image information extraction method; and, the resolution of the local descriptor vector output by the descriptor generation network when training the image information extraction model is: N times the resolution of the local descriptor vector output by the descriptor generation network when executing the image information extraction method.

[0078] Accordingly, as an example, when the resolution of the original feature map extracted when training the image information extraction model and when executing the image information extraction method is the same, the resolution of the feature point detection result image obtained when training the image information extraction model is: N times the resolution of the feature point detection result image obtained when executing the image information extraction method; and the resolution of the local descriptor result obtained when training the image information extraction model is: N times the resolution of the local descriptor result obtained when executing the image information extraction method. In other words, higher-resolution descriptor supervision and higher-resolution feature point supervision are performed during training relative to prediction.

[0079] N is a multiple of 2 that is greater than 0. In fact, the value of N is determined by the multiple of the original feature map magnified by the upsampling process performed by the upsampling module. In other words, the original feature map is enlarged N times through the upsampling process.

[0080] As an example, the image information extraction model also includes: a global descriptor acquisition module. The image information extraction method according to the exemplary embodiment of the present disclosure may also include: inputting the original feature map into the global descriptor acquisition module to obtain a global descriptor result of the target image.

[0081] As an example, the global descriptor acquisition module may compress the original feature map into a one-dimensional vector, and perform a linear transformation (e.g., through a fully connected layer) on the compressed one-dimensional vector to obtain a global descriptor result of the target image. It should be understood that the global descriptor result of the target image may also be obtained based on the original feature map in other ways, and the present disclosure is not limited to this.

[0082] It should be understood that the local descriptor results and feature point detection results of the acquired target image can be applied to image feature matching scenarios, and it should be understood that they can also be applied to other scenarios, and the present disclosure does not limit this.

[0083] Figure 2A flow chart of a training method for an image information extraction model according to an exemplary embodiment of the present disclosure is shown. The image information extraction model includes: a feature extraction module, a feature point detection network, and a descriptor generation network.

[0084] Reference Figure 2 In step S201, a training sample is obtained, wherein the training sample includes: a first sample image with annotations and a second sample image with annotations, wherein the second sample image is obtained by transforming the first sample image, and the annotation includes: the position of the feature point in the image.

[0085] As an example, the second sample image may be obtained by randomly performing a transformation operation such as rotation / distortion / affine transformation on the first sample image.

[0086] As an example, the first sample image may be an original image or an image portion including a specific object cropped from the original image.

[0087] In step S202, for each sample image in the first sample image and the second sample image, the sample image is input into the feature extraction network in the feature extraction module, and the upsampling module in the feature extraction module is used to upsample the original feature map of the sample output by the feature extraction network to obtain an intermediate feature map of the sample.

[0088] As an example, the feature extraction network may include multiple convolutional neural networks for downsampling processing. As an example, each convolutional neural network for downsampling processing may use separable convolution and have an inverted residual spindle structure.

[0089] As an example, the upsampling process may be: a fixed interpolation upsampling process.

[0090] As an example, the resolution of the sample intermediate feature map is N times that of the sample original feature map.

[0091] In step S203, the sample feature map extracted by the feature extraction module is input into the feature point detection network and the descriptor generation network respectively to obtain the feature point detection result vector output by the feature point detection network and the local descriptor vector output by the descriptor generation network, wherein the sample feature map extracted by the feature extraction module is the sample intermediate feature map.

[0092] Specifically, the sample feature map of the first sample image extracted by the feature extraction module is input into the feature point detection network and the descriptor generation network respectively, so as to obtain the feature point detection result vector of the first sample image output by the feature point detection network and the local descriptor vector of the first sample image output by the descriptor generation network. The sample feature map of the second sample image extracted by the feature extraction module is input into the feature point detection network and the descriptor generation network respectively, so as to obtain the feature point detection result vector of the second sample image output by the feature point detection network and the local descriptor vector of the second sample image output by the descriptor generation network.

[0093] In step S204, based on the feature point detection result vector, a feature point detection result image with a resolution N times the original resolution of the sample image is obtained, where N is an even number.

[0094] Specifically, based on the feature point detection result vector of the first sample image, a feature point detection result image with a resolution N times the original resolution of the first sample image (i.e., the feature point detection result image of the first sample image) is obtained. Based on the feature point detection result vector of the second sample image, a feature point detection result image with a resolution N times the original resolution of the second sample image (i.e., the feature point detection result image of the second sample image) is obtained.

[0095] As an example, the feature point detection result vector may be passed through a convolutional neural network and reorganized among multiple channels to obtain a feature point detection result image having a resolution N times the original resolution of the sample image.

[0096] In step S205, based on the local descriptor vector, a local descriptor result with a resolution N times the original resolution of the sample image is obtained.

[0097] Specifically, based on the local descriptor vector of the first sample image, a local descriptor result having a resolution N times the original resolution of the first sample image (i.e., the local descriptor result of the first sample image) is obtained. Based on the local descriptor vector of the second sample image, a local descriptor result having a resolution N times the original resolution of the second sample image (i.e., the local descriptor result of the second sample image) is obtained.

[0098] As an example, the local descriptor vector may be subjected to an interpolation up-sampling process to obtain a local descriptor result having a resolution N times the original resolution of the sample image.

[0099] In step S206, the image information extraction model is trained based on the feature point detection result images, local descriptor results, and annotations of the first sample image and the second sample image.

[0100] As an example, a first loss function may be determined based on the local descriptor result of the first sample image and the local descriptor result of the second sample image; a second loss function may be determined based on the feature point detection result image of the first sample image and the annotation of the first sample image, and the feature point detection result image of the second sample image and the annotation of the second sample image; and a total loss function may be determined based on the first loss function and the second loss function. Then, the image information extraction model may be trained by adjusting the parameters of the feature extraction module, the feature point detection network, and the descriptor generation network according to the total loss function.

[0101] In the present disclosure, when performing image information extraction of a target image based on the trained image information extraction model, the original feature map is used as the feature map extracted by the feature extraction module. In other words, when performing image information extraction based on the trained image information extraction model, the original feature map output by the feature extraction network does not need to be up-sampled by the up-sampling module and is directly input into the feature point detection network and the descriptor generation network respectively.

[0102] Accordingly, as an example, when image information extraction of the target image is performed based on the trained image information extraction model, the resolution of the feature point detection result image of the target image is the original resolution size of the target image; when image information extraction of the target image is performed based on the trained image information extraction model, the resolution of the local descriptor result of the target image is the original resolution size of the target image.

[0103] As an example, the training method of the image information extraction model according to the exemplary embodiment of the present disclosure may also include: inputting the sample original feature map of the first sample image into the global descriptor acquisition module to obtain the global descriptor result of the first sample image, and inputting the sample original feature map of the second sample image into the global descriptor acquisition module to obtain the global descriptor result of the second sample image; based on the global descriptor result of the first sample image and the global descriptor result of the second sample image, determining a third loss function. Wherein, based on the first loss function, the second loss function, and the third loss function, the total loss function is determined; and the image information extraction model is trained by adjusting the parameters of the feature extraction module, the feature point detection network, the descriptor generation network, and the global descriptor acquisition module according to the total loss function.

[0104] As an example, the sample original feature map of the first sample image can be input into the global descriptor acquisition module, and the global descriptor acquisition module compresses the sample original feature map of the first sample image into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain the global descriptor result of the first sample image; the sample original feature map of the second sample image can be input into the global descriptor acquisition module, and the global descriptor acquisition module compresses the sample original feature map of the second sample image into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain the global descriptor result of the second sample image.

[0105] The present disclosure takes into account that the feature point detection branch can obtain the true value, but the descriptor generation branch is difficult to obtain the true value result (ground truth), so it is proposed to train the descriptor (including global descriptors and local descriptors) in a self-supervised manner. For an input image, random rotation / distortion / affine transformation operations are performed to obtain another image, and the positions of the feature points on the two images can be obtained. The two images are passed through the network respectively to obtain the descriptors of the initial two images, and the loss of these two descriptors is calculated, and back-propagation is performed to train the descriptors (the definition of the descriptor is that for feature points on similar images, the corresponding descriptor distance is the minimum).

[0106] It should be understood that a plurality of training samples may be used to train the image information extraction model.

[0107] Figure 3 A comparative schematic diagram showing an image information extraction method and an image information extraction model training method according to an exemplary embodiment of the present disclosure.

[0108] like Figure 3 As shown in the figure, during the training process, a 320*180 image input is used. After the features are extracted by the feature extraction network, a fixed interpolation upsampling is performed first, and then the feature point detection branch and the descriptor generation branch operations are performed to restore the image to a size of 640*360 for training.

[0109] In the process of using the trained image information extraction model for prediction, after obtaining the features extracted by the feature extraction network, remove the upsampling operation once, and directly perform the feature point detection branch and descriptor generation branch operations to restore the image to a size of 320*180 for prediction. Compared with the training process, the amount of calculation in the feature point detection and descriptor generation network part is reduced by 4 times (the image size is reduced by 4 times). Training on high-resolution images can enable the network to learn more detailed information, optimize network parameters, and solidify network parameters (i.e., trained network parameters) When predicting on low-resolution images, the amount of calculation will be greatly reduced due to the 4-fold reduction in image size.

[0110] Figure 4 The structural block diagram of an image information extraction device according to an exemplary embodiment of the present disclosure is shown. The image information extraction device performs prediction based on a trained image information extraction model, and the image information extraction model includes: a feature extraction module, a feature point detection network, and a descriptor generation network.

[0111] like Figure 4 As shown, the image information extraction device 10 according to an exemplary embodiment of the present disclosure includes: a feature extraction unit 101, an information extraction unit 102, a feature point acquisition unit 103, and a local descriptor acquisition unit 104.

[0112] Specifically, the feature extraction unit 101 is configured to input the target image into the feature extraction network in the feature extraction module to obtain the original feature map output by the feature extraction network.

[0113] The information extraction unit 102 is configured to input the feature map extracted by the feature extraction module into the feature point detection network and the descriptor generation network respectively, to obtain the feature point detection result vector output by the feature point detection network and the local descriptor vector output by the descriptor generation network, wherein the feature map extracted by the feature extraction module is the original feature map.

[0114] The feature point acquisition unit 103 is configured to obtain a feature point detection result image with a resolution equal to the original resolution of the target image based on the feature point detection result vector, wherein the pixel value of each pixel in the feature point detection result image represents the probability that the pixel in the target image is a feature point.

[0115] The local descriptor obtaining unit 104 is configured to obtain a local descriptor result having a resolution equal to the original resolution of the target image based on the local descriptor vector.

[0116] When the training device (for example, the training device 20) trains the image information extraction model, the upsampling module in the feature extraction module is used to upsample the original feature map of the sample output by the feature extraction network to obtain a sample intermediate feature map, wherein the sample intermediate feature map is used as the sample feature map extracted by the feature extraction module.

[0117] As an example, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample, where N is an even number; when training the image information extraction model, the resolution of the feature point detection result image of the sample image is N times the size of the original resolution of the sample image; when training the image information extraction model, the resolution of the local descriptor result of the sample image is N times the size of the original resolution of the sample image.

[0118] As an example, the upsampling process may be: a fixed interpolation upsampling process.

[0119] As an example, the image information extraction model may further include: a global descriptor acquisition module, wherein the image information extraction device 10 may further include: a global descriptor acquisition unit (not shown), the global descriptor acquisition unit being configured to input the original feature map into the global descriptor acquisition module to obtain a global descriptor result of the target image.

[0120] As an example, the global descriptor acquisition unit can be configured to input the original feature map into the global descriptor acquisition module, the global descriptor acquisition module compresses the original feature map into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the target image.

[0121] As an example, the feature extraction network may include multiple convolutional neural networks for downsampling processing, wherein each convolutional neural network for downsampling processing uses separable convolution and has an inverted residual spindle structure.

[0122] As an example, the feature point acquisition unit 103 may be configured to pass the feature point detection result vector through a convolutional neural network and perform multi-channel reorganization to obtain a feature point detection result image having a resolution equal to the original resolution of the target image.

[0123] As an example, the local descriptor acquisition unit 104 may be configured to perform an interpolation upsampling process on the local descriptor vector to obtain a local descriptor result having a resolution equal to the original resolution of the target image.

[0124] As an example, the target image may be an original image or an image portion including a specific object cropped from the original image.

[0125] As an example, when the target image is an image portion including a specific object cropped from an original image, the image information extraction device 10 may further include: a target image acquisition unit (not shown), the target image acquisition unit being configured to input the original image into a target detection network to obtain an area where the specific object in the original image is located; crop the area from the original image as the target image; wherein the target detection network is an SSD network structure, and each convolutional neural network in the target detection network uses separable convolution.

[0126] Figure 5 The structural block diagram of the training device of the image information extraction model according to the exemplary embodiment of the present disclosure is shown. The image information extraction model includes: a feature extraction module, a feature point detection network, and a descriptor generation network.

[0127] like Figure 5 As shown, the image information extraction model 20 according to the exemplary embodiment of the present disclosure includes: a training sample acquisition unit 201, a feature extraction unit 202, an information extraction unit 203, a feature point acquisition unit 204, a local descriptor acquisition unit 205, and a training unit 206.

[0128] Specifically, the training sample acquisition unit 201 is configured to acquire training samples, wherein the training samples include: a first sample image with annotations and a second sample image with annotations, wherein the second sample image is obtained by transforming the first sample image, and the annotations include: the positions of feature points in the image.

[0129] The feature extraction unit 202 is configured to input the sample image into the feature extraction network in the feature extraction module for each sample image in the first sample image and the second sample image, and use the upsampling module in the feature extraction module to upsample the original feature map of the sample output by the feature extraction network to obtain an intermediate feature map of the sample.

[0130] The information extraction unit 203 is configured to input the sample feature map extracted by the feature extraction module into the feature point detection network and the descriptor generation network respectively, to obtain the feature point detection result vector output by the feature point detection network and the local descriptor vector output by the descriptor generation network, wherein the sample feature map extracted by the feature extraction module is the sample intermediate feature map.

[0131] The feature point acquisition unit 204 is configured to obtain a feature point detection result image having a resolution N times the original resolution of the sample image based on the feature point detection result vector.

[0132] The local descriptor obtaining unit 205 is configured to obtain a local descriptor result having a resolution N times the original resolution of the sample image based on the local descriptor vector.

[0133] The training unit 206 is configured to train the image information extraction model based on the feature point detection result images, local descriptor results, and annotations of the first sample image and the second sample image.

[0134] When performing image information extraction based on the trained image information extraction model, the image information extraction device (for example, the image information extraction device 10) uses the original feature map as the feature map extracted by the feature extraction module; wherein N is an even number.

[0135] As an example, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample; when image information extraction of the target image is performed based on the trained image information extraction model, the resolution of the feature point detection result image of the target image is the original resolution size of the target image; when image information extraction of the target image is performed based on the trained image information extraction model, the resolution of the local descriptor result of the target image is the original resolution size of the target image.

[0136] As an example, the image information extraction model may further include: a global descriptor acquisition module, wherein the training device 20 may further include: a global descriptor acquisition unit (not shown), the global descriptor acquisition unit being configured to input the sample original feature map of the first sample image into the global descriptor acquisition module to obtain the global descriptor result of the first sample image, and input the sample original feature map of the second sample image into the global descriptor acquisition module to obtain the global descriptor result of the second sample image; wherein the training unit 206 may be configured to: determine the first sample image based on the local descriptor result and the local descriptor result of the second sample image Loss function; determine a second loss function based on the feature point detection result image of the first sample image and the annotation of the first sample image, and the feature point detection result image of the second sample image and the annotation of the second sample image; determine a third loss function based on the global descriptor result of the first sample image and the global descriptor result of the second sample image; determine a total loss function based on the first loss function, the second loss function, and the third loss function; train the image information extraction model by adjusting the parameters of the feature extraction module, the feature point detection network, the descriptor generation network, and the global descriptor acquisition module according to the total loss function.

[0137] As an example, the global descriptor acquisition unit may be configured to input the sample original feature map of the first sample image into the global descriptor acquisition module, the global descriptor acquisition module compresses the sample original feature map of the first sample image into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the first sample image; input the sample original feature map of the second sample image into the global descriptor acquisition module, the global descriptor acquisition module compresses the sample original feature map of the second sample image into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the second sample image.

[0138] As an example, the feature extraction network may include multiple convolutional neural networks for downsampling processing, wherein each convolutional neural network for downsampling processing uses separable convolution and has an inverted residual spindle structure.

[0139] As an example, the upsampling process may be: a fixed interpolation upsampling process.

[0140] As an example, the feature point acquisition unit 204 may be configured to pass the feature point detection result vector through a convolutional neural network and perform multi-channel reorganization to obtain a feature point detection result image with a resolution N times the original resolution of the sample image.

[0141] As an example, the local descriptor acquisition unit 205 may be configured to perform an interpolation up-sampling process on the local descriptor vector to obtain a local descriptor result having a resolution N times the original resolution of the sample image.

[0142] As an example, the first sample image may be an original image or an image portion including a specific object cropped from the original image.

[0143] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the relevant method, and will not be elaborated here.

[0144] In addition, it should be understood that the various units in the image information extraction device 10 and the image information extraction model training device 20 according to the exemplary embodiment of the present disclosure can be implemented as hardware components and / or software components. Those skilled in the art can implement various units, for example, using a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC) according to the processing performed by the defined various units.

[0145] Figure 6 FIG. 1 is a block diagram showing a structure of an electronic device according to an exemplary embodiment of the present disclosure. Figure 6 The electronic device 30 includes: at least one memory 301 and at least one processor 302, wherein the at least one memory 301 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 302, the image information extraction method and / or the image information extraction model training method described in the above exemplary embodiments are executed.

[0146] As an example, the electronic device 30 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 30 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 30 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0147] In the electronic device 30, the processor 302 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 302 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0148] The processor 302 may execute instructions or codes stored in the memory 301, wherein the memory 301 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0149] The memory 301 may be integrated with the processor 302, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 301 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 301 and the processor 302 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 302 can read files stored in the memory.

[0150] In addition, the electronic device 30 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 30 may be connected to each other via a bus and / or a network.

[0151] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the image information extraction method and / or the image information extraction model training method as described in the above exemplary embodiments. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0152] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by at least one processor to complete the image information extraction method and / or the image information extraction model training method as described in the above exemplary embodiments.

[0153] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0154] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for extracting image information, characterized in that: The image information extraction method is performed based on a trained image information extraction model, wherein the image information extraction model includes: a feature extraction module, a feature point detection network and a descriptor generation network, wherein the image information extraction method includes: Inputting the target image into the feature extraction network in the feature extraction module to obtain the original feature map output by the feature extraction network; Inputting the feature map extracted by the feature extraction module into the feature point detection network and the descriptor generation network respectively, obtaining a feature point detection result vector output by the feature point detection network and a local descriptor vector output by the descriptor generation network, wherein the feature map extracted by the feature extraction module is the original feature map; Based on the feature point detection result vector, a feature point detection result image with a resolution equal to the original resolution of the target image is obtained, wherein the pixel value of each pixel in the feature point detection result image represents the probability that the pixel in the target image is a feature point; Based on the local descriptor vector, a local descriptor result having a resolution equal to the original resolution of the target image is obtained; When training the image information extraction model, the upsampling module in the feature extraction module is used to upsample the original feature map of the sample output by the feature extraction network to obtain an intermediate feature map of the sample, wherein the intermediate feature map of the sample is used as the sample feature map extracted by the feature extraction module; Wherein, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample, where N is an even number; When training the image information extraction model, the resolution of the feature point detection result image of the sample image obtained is N times the original resolution of the sample image; When training the image information extraction model, the resolution of the local descriptor result of the sample image obtained is N times the original resolution of the sample image.

2. The image information extraction method according to claim 1, characterized in that: The up-sampling process is: fixed interpolation up-sampling process.

3. The image information extraction method according to claim 1, characterized in that: The image information extraction model further includes: a global descriptor acquisition module, wherein the image information extraction method further includes: The original feature map is input into the global descriptor acquisition module to obtain the global descriptor result of the target image.

4. The image information extraction method according to claim 3, characterized in that: The step of inputting the original feature map into the global descriptor acquisition module to obtain the global descriptor result of the target image comprises: The original feature map is input into the global descriptor acquisition module, and the global descriptor acquisition module compresses the original feature map into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the target image.

5. The image information extraction method according to claim 1, characterized in that: The feature extraction network includes a plurality of convolutional neural networks for downsampling processing. Among them, each convolutional neural network used for downsampling processing uses separable convolution and has an inverted residual spindle structure.

6. The image information extraction method according to claim 1, characterized in that: The step of obtaining a feature point detection result image with a resolution equal to the original resolution of the target image based on the feature point detection result vector comprises: The feature point detection result vector is passed through a convolutional neural network and reorganized among multiple channels to obtain a feature point detection result image with a resolution equal to the original resolution of the target image.

7. The image information extraction method according to claim 1, characterized in that: The step of obtaining a local descriptor result having a resolution equal to the original resolution of the target image based on the local descriptor vector comprises: The local descriptor vector is interpolated and up-sampled to obtain a local descriptor result having a resolution equal to the original resolution of the target image.

8. The image information extraction method according to claim 1, characterized in that: The target image is the original image or an image portion cropped from the original image and including a specific object.

9. The image information extraction method according to claim 8, characterized in that: When the target image is an image portion including a specific object cut out from an original image, the image information extraction method further includes: Input the original image into the target detection network to obtain the area where the specific object is located in the original image; Cut out the region from the original image as the target image; Among them, the target detection network is an SSD network structure, and each convolutional neural network in the target detection network uses separable convolution.

10. A training method for an image information extraction model, characterized in that: The image information extraction model includes: a feature extraction module, a feature point detection network, and a descriptor generation network, and the training method includes: Acquire a training sample, wherein the training sample includes: a first sample image with annotations and a second sample image with annotations, wherein the second sample image is obtained by transforming the first sample image, and the annotations include: positions of feature points in the image; For each sample image in the first sample image and the second sample image, the sample image is input into the feature extraction network in the feature extraction module, and an upsampling module in the feature extraction module is used to upsample the sample original feature map output by the feature extraction network to obtain a sample intermediate feature map; Inputting the sample feature map extracted by the feature extraction module into the feature point detection network and the descriptor generation network respectively, obtaining the feature point detection result vector output by the feature point detection network and the local descriptor vector output by the descriptor generation network, wherein the sample feature map extracted by the feature extraction module is the sample intermediate feature map; Based on the feature point detection result vector, a feature point detection result image with a resolution N times the original resolution of the sample image is obtained; Based on the local descriptor vector, a local descriptor result with a resolution N times the original resolution of the sample image is obtained; Training the image information extraction model based on the feature point detection result images, local descriptor results, and annotations of the first sample image and the second sample image; Wherein, when performing image information extraction based on the trained image information extraction model, the original feature map is used as the feature map extracted by the feature extraction module; Wherein, N is an even number; Wherein, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample; When performing image information extraction of a target image based on the trained image information extraction model, the resolution of the feature point detection result image of the target image obtained is the original resolution size of the target image; When image information extraction of a target image is performed based on the trained image information extraction model, the resolution of the local descriptor result of the target image obtained is the original resolution size of the target image.

11. The training method according to claim 10, characterized in that: The image information extraction model further includes: a global descriptor acquisition module, wherein the training method further includes: Inputting the sample original feature map of the first sample image into the global descriptor acquisition module to obtain the global descriptor result of the first sample image, and inputting the sample original feature map of the second sample image into the global descriptor acquisition module to obtain the global descriptor result of the second sample image; The step of training the image information extraction model based on the feature point detection result images, local descriptor results, and annotations of the first sample image and the second sample image includes: Determine a first loss function based on a local descriptor result of the first sample image and a local descriptor result of the second sample image; Determine a second loss function based on the feature point detection result image of the first sample image and the annotation of the first sample image, and the feature point detection result image of the second sample image and the annotation of the second sample image; Determining a third loss function based on the global descriptor result of the first sample image and the global descriptor result of the second sample image; Determine a total loss function based on the first loss function, the second loss function, and the third loss function; The image information extraction model is trained by adjusting the parameters of the feature extraction module, the feature point detection network, the descriptor generation network and the global descriptor acquisition module according to the total loss function.

12. The training method according to claim 11, characterized in that: The steps of inputting the sample original feature map of the first sample image into the global descriptor acquisition module to obtain the global descriptor result of the first sample image, and inputting the sample original feature map of the second sample image into the global descriptor acquisition module to obtain the global descriptor result of the second sample image include: Inputting the sample original feature map of the first sample image into the global descriptor acquisition module, the global descriptor acquisition module compresses the sample original feature map of the first sample image into a one-dimensional vector, and linearly transforms the compressed one-dimensional vector to obtain a global descriptor result of the first sample image; The sample original feature map of the second sample image is input into the global descriptor acquisition module, and the global descriptor acquisition module compresses the sample original feature map of the second sample image into a one-dimensional vector, and linearly transforms the compressed one-dimensional vector to obtain a global descriptor result of the second sample image.

13. The training method according to claim 10, characterized in that: The feature extraction network includes a plurality of convolutional neural networks for downsampling processing. Among them, each convolutional neural network used for downsampling processing uses separable convolution and has an inverted residual spindle structure.

14. The training method according to claim 10, characterized in that: The up-sampling process is: fixed interpolation up-sampling process.

15. The training method according to claim 10, characterized in that: The step of obtaining a feature point detection result image with a resolution N times the original resolution of the sample image based on the feature point detection result vector comprises: The feature point detection result vector is passed through a convolutional neural network and reorganized among multiple channels to obtain a feature point detection result image with a resolution N times the original resolution of the sample image.

16. The training method according to claim 10, characterized in that: The step of obtaining a local descriptor result having a resolution N times the original resolution of the sample image based on the local descriptor vector comprises: The local descriptor vector is interpolated and up-sampled to obtain a local descriptor result with a resolution N times the original resolution of the sample image.

17. The training method according to claim 10, characterized in that: The first sample image is an original image or an image portion cut out from the original image and including a specific object.

18. An image information extraction device, characterized in that: The image information extraction device performs operations based on a trained image information extraction model, wherein the image information extraction model includes: a feature extraction module, a feature point detection network, and a descriptor generation network, wherein the image information extraction device includes: A feature extraction unit is configured to input a target image into a feature extraction network in the feature extraction module to obtain an original feature map output by the feature extraction network; An information extraction unit is configured to input the feature map extracted by the feature extraction module into the feature point detection network and the descriptor generation network respectively, to obtain a feature point detection result vector output by the feature point detection network and a local descriptor vector output by the descriptor generation network, wherein the feature map extracted by the feature extraction module is the original feature map; A feature point acquisition unit is configured to obtain a feature point detection result image having a resolution equal to the original resolution of the target image based on the feature point detection result vector, wherein the pixel value of each pixel in the feature point detection result image represents the probability that the pixel in the target image is a feature point; A local descriptor acquisition unit is configured to obtain a local descriptor result having a resolution equal to the original resolution of the target image based on the local descriptor vector; When the training device trains the image information extraction model, the upsampling module in the feature extraction module is used to upsample the original feature map of the sample output by the feature extraction network to obtain an intermediate feature map of the sample, wherein the intermediate feature map of the sample is used as the sample feature map extracted by the feature extraction module; Wherein, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample, where N is an even number; When training the image information extraction model, the resolution of the feature point detection result image of the sample image obtained is N times the original resolution of the sample image; When training the image information extraction model, the resolution of the local descriptor result of the sample image obtained is N times the original resolution of the sample image.

19. The image information extraction device according to claim 18, characterized in that: The up-sampling process is: fixed interpolation up-sampling process.

20. The image information extraction device according to claim 18, characterized in that: The image information extraction model further includes: a global descriptor acquisition module, wherein the image information extraction device further includes: The global descriptor acquisition unit is configured to input the original feature map into the global descriptor acquisition module to obtain a global descriptor result of the target image.

21. The image information extraction device according to claim 20, characterized in that: The global descriptor acquisition unit is configured to input the original feature map into the global descriptor acquisition module, and the global descriptor acquisition module compresses the original feature map into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the target image.

22. The image information extraction device according to claim 18, characterized in that: The feature extraction network includes a plurality of convolutional neural networks for downsampling processing. Among them, each convolutional neural network used for downsampling processing uses separable convolution and has an inverted residual spindle structure.

23. The image information extraction device according to claim 18, characterized in that: The feature point acquisition unit is configured to pass the feature point detection result vector through a convolutional neural network and perform multi-channel reorganization to obtain a feature point detection result image with a resolution equal to the original resolution of the target image.

24. The image information extraction device according to claim 18, characterized in that: The local descriptor acquisition unit is configured to perform interpolation upsampling processing on the local descriptor vector to obtain a local descriptor result with a resolution equal to the original resolution of the target image.

25. The image information extraction device according to claim 18, characterized in that: The target image is the original image or an image portion cropped from the original image and including a specific object.

26. The image information extraction device according to claim 25, characterized in that: When the target image is an image portion including a specific object cut out from an original image, the image information extraction device further includes: A target image acquisition unit is configured to input the original image into the target detection network to obtain the area where the specific object is located in the original image; and to crop the area from the original image as the target image; Among them, the target detection network is an SSD network structure, and each convolutional neural network in the target detection network uses separable convolution.

27. A training device for an image information extraction model, characterized in that: The image information extraction model includes: a feature extraction module, a feature point detection network, and a descriptor generation network, and the training device includes: A training sample acquisition unit is configured to acquire training samples, wherein the training samples include: a first sample image with annotations and a second sample image with annotations, wherein the second sample image is obtained by transforming the first sample image, and the annotations include: positions of feature points in the image; a feature extraction unit configured to input the sample image into a feature extraction network in the feature extraction module for each sample image in the first sample image and the second sample image, and perform upsampling processing on the sample original feature map output by the feature extraction network by using an upsampling module in the feature extraction module to obtain a sample intermediate feature map; An information extraction unit is configured to input the sample feature map extracted by the feature extraction module into the feature point detection network and the descriptor generation network respectively, to obtain a feature point detection result vector output by the feature point detection network and a local descriptor vector output by the descriptor generation network, wherein the sample feature map extracted by the feature extraction module is the sample intermediate feature map; A feature point acquisition unit is configured to obtain a feature point detection result image with a resolution N times the original resolution of the sample image based on the feature point detection result vector; A local descriptor acquisition unit is configured to obtain a local descriptor result having a resolution N times the original resolution of the sample image based on the local descriptor vector; A training unit, configured to train the image information extraction model based on the feature point detection result images, local descriptor results, and annotations of the first sample image and the second sample image; Wherein, when the image information extraction device performs image information extraction based on the trained image information extraction model, the original feature map is used as the feature map extracted by the feature extraction module; Wherein, N is an even number; Wherein, the resolution of the intermediate feature map of the sample is N times that of the original feature map of the sample; When performing image information extraction of a target image based on the trained image information extraction model, the resolution of the feature point detection result image of the target image obtained is the original resolution size of the target image; When image information extraction of a target image is performed based on the trained image information extraction model, the resolution of the local descriptor result of the target image obtained is the original resolution size of the target image.

28. The training device according to claim 27, characterized in that The image information extraction model further includes: a global descriptor acquisition module, wherein the training device further includes: a global descriptor acquisition unit, configured to input the sample original feature map of the first sample image into the global descriptor acquisition module to obtain a global descriptor result of the first sample image, and input the sample original feature map of the second sample image into the global descriptor acquisition module to obtain a global descriptor result of the second sample image; Wherein, the training unit is configured to: determine a first loss function based on the local descriptor result of the first sample image and the local descriptor result of the second sample image; determine a second loss function based on the feature point detection result image of the first sample image and the annotation of the first sample image, and the feature point detection result image of the second sample image and the annotation of the second sample image; determine a third loss function based on the global descriptor result of the first sample image and the global descriptor result of the second sample image; determine a total loss function based on the first loss function, the second loss function, and the third loss function; and train the image information extraction model by adjusting the parameters of the feature extraction module, the feature point detection network, the descriptor generation network, and the global descriptor acquisition module according to the total loss function.

29. The training device according to claim 28, characterized in that The global descriptor acquisition unit is configured to input the sample original feature map of the first sample image into the global descriptor acquisition module, the global descriptor acquisition module compresses the sample original feature map of the first sample image into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the first sample image; input the sample original feature map of the second sample image into the global descriptor acquisition module, the global descriptor acquisition module compresses the sample original feature map of the second sample image into a one-dimensional vector, and performs a linear transformation on the compressed one-dimensional vector to obtain a global descriptor result of the second sample image.

30. The training device according to claim 27, characterized in that The feature extraction network includes a plurality of convolutional neural networks for downsampling processing. Among them, each convolutional neural network used for downsampling processing uses separable convolution and has an inverted residual spindle structure.

31. The training device according to claim 27, characterized in that The up-sampling process is: fixed interpolation up-sampling process.

32. The training device according to claim 27, characterized in that The feature point acquisition unit is configured to pass the feature point detection result vector through a convolutional neural network and perform multi-channel reorganization to obtain a feature point detection result image with a resolution N times the original resolution of the sample image.

33. The training device according to claim 27, characterized in that The local descriptor acquisition unit is configured to perform interpolation up-sampling processing on the local descriptor vector to obtain a local descriptor result with a resolution N times the original resolution of the sample image.

34. The training device according to claim 27, characterized in that The first sample image is an original image or an image portion cut out from the original image and including a specific object.

35. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the image information extraction method as described in any one of claims 1 to 9 and / or the image information extraction model training method as described in any one of claims 10 to 17.

36. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image information extraction method as described in any one of claims 1 to 9 and / or the image information extraction model training method as described in any one of claims 10 to 17.

37. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the image information extraction method according to any one of claims 1 to 9 and / or the image information extraction model training method according to any one of claims 10 to 17 are implemented.

Images