Information extraction method, device, electronic device and computer readable storage medium

Abstract

The embodiment of the present application discloses an information extraction method, device, electronic device and computer-readable storage medium, the method comprising: obtaining a target image to be processed; extracting key point information of the target image through a key point extraction model, and extracting description information of the target image through a descriptor extraction model; determining the descriptor information corresponding to the key point information from the description information according to the description offset information corresponding to the descriptor extraction model; generating image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information; wherein the key point extraction model and the descriptor extraction model are trained separately. The training of the key point extraction model and the descriptor extraction model separately helps to improve the training efficiency of the key point extraction model and the descriptor extraction model, and the solution of processing the key point information and the description information separately can more easily meet the needs of downstream tasks.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and specifically to an information extraction method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Information extraction is particularly important in the field of artificial intelligence. For example, extracting key point information or feature point information of an image is the basis of computer vision algorithms and helps to achieve tasks such as spatial positioning, target analysis and detection. Among them, feature point information includes key point information and the descriptor information of the key point.

[0003] Currently, there are feature point extraction algorithms available, such as SIFT and ORB. SIFT runs slowly on mobile devices, and although ORB runs fast, its effect is poor and cannot meet the needs of complex 3D vision tasks. At the same time, with the continuous development of deep learning technology, in recent years, algorithms that use data-driven methods and deep learning to extract feature points have emerged, such as SuperPoint, SiLK, DISK, etc.

[0004] However, the current deep learning algorithms are mainly aimed at extracting feature points, which cannot meet the needs of extracting key point information separately. They are not suitable for downstream tasks of analyzing key point information and are difficult to meet the needs of downstream tasks of information extraction. Summary of the invention

[0005] The embodiments of the present application provide an information extraction method, device, electronic device and computer-readable storage medium, which can extract key point information separately when extracting feature point information, making it easier to meet the needs of downstream tasks.

[0006] In a first aspect, an embodiment of the present application provides an information extraction method, the method comprising:

[0007] Obtaining a target image to be processed;

[0008] Extracting key point information of the target image through a key point extraction model, and extracting description information of the target image through a descriptor extraction model;

[0009] Determining the descriptor information corresponding to the key point information from the description information according to the description offset information corresponding to the descriptor extraction model;

[0010] Generate image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information;

[0011] Wherein, the key point extraction model and the descriptor extraction model are trained separately.

[0012] In a second aspect, an embodiment of the present application further provides an information extraction device, the device comprising:

[0013] An acquisition module, used for acquiring a target image to be processed;

[0014] An extraction module, used to extract key point information of the target image through a key point extraction model, and to extract description information of the target image through a descriptor extraction model;

[0015] A determination module, configured to determine the descriptor information corresponding to the key point information from the description information according to the description offset information corresponding to the descriptor extraction model;

[0016] A generating module, used for generating image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information;

[0017] Wherein, the key point extraction model and the descriptor extraction model are trained separately.

[0018] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and which implements the steps in the above-mentioned information extraction method when the computer program is executed by the processor.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned information extraction method are implemented.

[0020] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0021] The embodiment of the present application obtains a target image to be processed, extracts key point information of the target image through a key point extraction model, and extracts description information of the target image through a descriptor extraction model, determines descriptor information corresponding to the key point information from the description information according to description offset information corresponding to the descriptor extraction model, and generates image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information. The key point extraction model and the descriptor extraction model are trained separately.

[0022] The key point information and description information of the target image are extracted, and the corresponding descriptor information is determined from the description information according to the key point information, so as to obtain the feature point information including the key point information and the descriptor information corresponding to the key point information.

[0023] Among them, by training the key point extraction model and the descriptor extraction model separately, it is helpful to use the key point extraction model alone to extract the key point information of the target image, and realize the extraction of key point information, and it is also helpful to use the descriptor extraction model alone to extract the description information of the target image. Such a solution of processing key point information and description information separately is helpful for processing downstream tasks of key point information analysis or description information analysis, that is, it is easier to meet the needs of downstream tasks. Among them, training the key point extraction model and the descriptor extraction model separately also helps to improve the training efficiency of the key point extraction model and the descriptor extraction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a schematic diagram of a scenario in which an extended reality device provided in an embodiment of the present application executes the information extraction method;

[0026] Figure 2 It is a flowchart of the information extraction method provided in the embodiment of the present application;

[0027] Figure 3 is a flow chart of a key point extraction model obtained through training provided in an embodiment of the present application;

[0028] Figure 4 is a structural diagram of a key point extraction model provided in an embodiment of the present application;

[0029] Figure 5 is a structural diagram of a descriptor extraction model provided in an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of the structure of an information extraction device provided in an embodiment of the present application;

[0031] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0032] Description of Figure Numbers:

[0033] 10-extended reality device; 201-acquisition module; 202-extraction module; 203-determination module; 204-generation module; 301-processor; 302-memory; 303-power supply; 304-input unit; 305-camera. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0035] The embodiments of the present application provide an information extraction method, device, electronic device and computer-readable storage medium. Specifically, the embodiments of the present application provide an information extraction device suitable for electronic devices, the electronic devices include but are not limited to mobile phones, tablet computers, laptop computers, televisions or extended reality devices with computing processing units, wherein the extended reality devices include but are not limited to head-mounted displays and wearable glasses, and the extended reality devices can be integrated extended reality devices built into the computing processing unit, or they can be split extended reality devices externally connected to the computing processing unit. Among them, the extended reality devices include but are not limited to airborne optical display systems, i.e., head-up display systems, applied to vehicles such as aircraft, cars, and ships, for example, AR-HUD (Augmented reality hud-up display) mounted on smart connected cars, handheld mobile devices such as mobile phones, desktop computers, laptop computers, tablets, etc., wearable near-eye display systems such as head-mounted displays, smart glasses, etc. When the extended reality device is a wearable head-mounted display or smart glasses, it can be an integrated extended reality device built into the computing processing unit, or it can be a split extended reality device externally connected to the computing processing unit.

[0036] When an extended reality device such as AR glasses displays virtual content in the real field of view, in order to achieve the enhanced effect, the displayed content needs to appear at a specified position. The premise is to know the relative position relationship between the glasses and the real field of view. This requires the use of real-time location tracking (SLAM) technology. Real-time location tracking based on vision is currently the mainstream in the field of extended reality. The most important thing about visual real-time location is to extract visual information quickly and accurately. Figure 1 , Figure 1 : is a schematic diagram of a scenario in which an extended reality device (such as optical see-through wearable smart AR glasses) provided in an embodiment of the present application executes the information extraction method, wherein the specific execution process of the extended reality device executing the information extraction method is as follows:

[0037] The extended reality device 10 obtains a target image to be processed, extracts key point information of the target image through a key point extraction model, and extracts description information of the target image through a descriptor extraction model, determines descriptor information corresponding to the key point information from the description information according to description offset information corresponding to the descriptor extraction model, and generates image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information. The key point extraction model and the descriptor extraction model are trained separately.

[0038] It can be understood that the embodiments of the present application obtain the feature point information including the key point information and the descriptive information of the target image by extracting the key point information and the descriptive information, and determining the corresponding descriptor information from the description information based on the key point information, thereby obtaining the feature point information including the key point information and the descriptor information corresponding to the key point information.

[0039] Among them, by training the key point extraction model and the descriptor extraction model separately, it is helpful to use the key point extraction model alone to extract the key point information of the target image, and realize the extraction of key point information, and it is also helpful to use the descriptor extraction model alone to extract the description information of the target image. Such a solution of processing key point information and description information separately is helpful for processing downstream tasks of key point information analysis or description information analysis, that is, it is easier to meet the needs of downstream tasks. Among them, training the key point extraction model and the descriptor extraction model separately also helps to improve the training efficiency of the key point extraction model and the descriptor extraction model.

[0040] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0041] See also Figure 2 , Figure 2 A flowchart of an information extraction method provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, the information extraction method is applied to an extended reality device, and the specific process of the method is as follows:

[0042] 101. Obtain a target image to be processed.

[0043] It can be understood that the target image is an image that needs to be subjected to information extraction, and the target image can be acquired from the scene after the extended reality device responds to a user instruction.

[0044] 102. The key point information of the target image is extracted by a key point extraction model, and the description information of the target image is extracted by a descriptor extraction model. The key point extraction model and the descriptor extraction model are trained separately.

[0045] It should be noted that feature point information refers to the information corresponding to feature points. Feature points are unique points in an image. They appear as obvious features in the image, such as corners, edges, textures, etc. Feature points are usually used for feature extraction and matching, and are the basis for image recognition and 3D reconstruction. In point cloud data, feature points refer to representative points that can reflect the local geometric structure of the point cloud.

[0046] Key point information is the information corresponding to the key point. Key points are a subset of feature points. They not only have the basic characteristics of feature points, but also have invariance, that is, they remain unchanged under the rotation, scaling and illumination changes of the image. Key points are usually used for image matching and pose estimation, such as feature point matching in SLAM (Simultaneous Localization and Mapping) systems.

[0047] Descriptor information is the information corresponding to the descriptor. Descriptors are vectors used to describe the characteristics of feature points or key points. They carry information about key points, such as position, size, direction, etc., and usually encode pixel information around the key points. Descriptors make it possible to match feature points between different images and act as "fingerprints" in the feature point matching process.

[0048] Among them, the feature points usually include key points and the descriptors corresponding to the key points. The descriptors are vectors that carry additional information about the key points, such as the position, size, direction, etc. of the key points. They usually encode the pixel information of the area around the key points.

[0049] It should be emphasized that the key point extraction model and the descriptor extraction model are trained separately, that is, the key point extraction model obtained after separate training can be used alone to extract the key point information in the target image, and the descriptor extraction model obtained after separate training can be used alone to extract the description information in the target image (the description information contains the descriptor information of each pixel in the target image).

[0050] Among them, the key point information extracted by the key point extraction model alone can be applied alone to the analysis and identification task of the key point information. Similarly, the description information extracted by the descriptor extraction model alone can also be applied alone to the analysis and identification task of the description information, making this solution easier to meet the needs of different downstream tasks. For example, the traditional feature point extraction method can only extract the feature point information of the image at the same time, but cannot extract the key point information or description information of the image alone. The embodiment of the present application obtains the key point extraction model and the descriptor extraction model by training separately, so that the key points and description information can be extracted by different models respectively, which helps to meet the downstream tasks of key point analysis or description information analysis, and it is easier to meet the needs of different downstream tasks.

[0051] 103. Determine the descriptor information corresponding to the key point information from the description information according to the description offset information corresponding to the descriptor extraction model.

[0052] It should be noted that the description information is for the entire target image. For example, the descriptor information of each pixel in the target image can be extracted by a descriptor extraction model.

[0053] The description offset information contains multiple coordinate information, which is obtained by the descriptor extraction model after training. The description offset information indicates which pixel descriptor information is extracted from the description information to generate the descriptor information corresponding to the key point information. For example, if the key point (i, j), the description offset information offset = (2, 5); then the descriptor information of the position (i+2, j+5) will be taken. Correspondingly, if the description offset information contains multiple coordinate information, the descriptor information of multiple positions will be extracted as the descriptor information of the key point.

[0054] 104. Generate image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information.

[0055] The image feature point information is the feature point information of the image. Since the feature point is composed of a key point and a descriptor corresponding to the key point, the feature point information of the image can be generated by combining the extracted key point information and the descriptor information corresponding to the key point information.

[0056] It can be understood that the embodiments of the present application obtain the feature point information including the key point information and the descriptive information of the target image by extracting the key point information and the descriptive information, and determining the corresponding descriptor information from the description information based on the key point information, thereby obtaining the feature point information including the key point information and the descriptor information corresponding to the key point information.

[0057] Among them, by training the key point extraction model and the descriptor extraction model separately, it is helpful to use the key point extraction model alone to extract the key point information of the target image, and realize the extraction of key point information, and it is also helpful to use the descriptor extraction model alone to extract the description information of the target image. Such a solution of processing key point information and description information separately is helpful for processing downstream tasks of key point information analysis or description information analysis, that is, it is easier to meet the needs of downstream tasks. Among them, training the key point extraction model and the descriptor extraction model separately also helps to improve the training efficiency of the key point extraction model and the descriptor extraction model.

[0058] Among them, in the embodiment of the present application, before applying the key point extraction model to extract key point information and using the descriptor extraction model to extract description information, it is necessary to first train the key point extraction model and the descriptor extraction model to improve the accuracy of the key point information and description information extraction. The training process of the key point extraction model and the training process of the descriptor extraction model will be described below, specifically:

[0059] Regarding the training part of the key point extraction model, the embodiment of the present application is divided into two stages, please refer to Figure 3 , Figure 3 This is a flowchart of the key point extraction model obtained by training provided in the embodiment of the present application, specifically:

[0060] Phase 1:

[0061] In order to improve the consistency of key point information extracted by the key point extraction model for the same key point under different visual conditions, the embodiment of the present application uses the same key point under different visual conditions to train the key point extraction model to ensure the repeatability of the key point information extracted under different viewing angles and different illuminations, that is, the consistency of the key point information extraction. That is, optionally, in some embodiments of the present application, the key point extraction model is obtained by the following steps:

[0062] Acquire at least one set of first sample image sets, the first sample image set comprising a sample original image and at least one sample transformed image obtained by performing a visual transformation on the sample original image, wherein the visual transformation comprises at least one of an illumination transformation and a perspective transformation;

[0063] For any first sample image set, extract key point information from each image in the first sample image set to obtain at least one sample key point information corresponding to each image;

[0064] For any sample key point information, obtaining extraction ratio information according to the number of pictures from which the sample key point information is extracted in the first sample image set and the total number of pictures corresponding to the first sample image set;

[0065] The key point extraction model is trained according to the key point information of each sample and the extraction ratio information corresponding to the key point information of each sample.

[0066] Among them, the sample original image is the collected original image, such as the image in the COCO dataset, and the sample transformed image is obtained after visual transformation based on the sample original image. The purpose is to obtain images of the same key points under different visual conditions such as different perspectives and different lighting environments. For example, the sample original image is subjected to illumination transformation and homography transformation to obtain the sample transformed image.

[0067] It should be noted that, in the embodiment of the present application, a known and mature key point extraction algorithm (such as SuperPoint) is used to extract key point information from each image in the first sample image set to obtain at least one sample key point information corresponding to each image. Wherein, each image in the first sample image set refers to a sample original image in the first sample image set and a sample transformed image obtained by visual transformation based on the sample original image.

[0068] The extraction ratio information refers to the ratio of the number of pictures that have extracted the key point information to the total number of pictures in the first sample image set. For example, if the first sample image set includes N pictures, for a certain key point, if the number of pictures that can repeatedly extract the key point is M, then the repeatability of the key point is P = M / N, wherein the closer P is to 1, the better the repeatability of the key point. On the contrary, if P is close to 0, it means that the key point is not robust to lighting and geometric transformations.

[0069] Correspondingly, key points whose repeatability meets a preset threshold can be screened out to train the model to obtain a key point extraction model. For example, for key points whose repeatability meets a preset threshold, the key points in the sample original image are annotated to obtain a pseudo-label of the key point (such as whether the key point exists or not, etc.). The sample original image is predicted by the model to be trained to obtain a response map of key point prediction, wherein the response map contains the probability value of each pixel point in the sample original image being a key point. Then, the cross-entropy loss can be calculated based on the predicted probability value corresponding to the pseudo-label and the response map, and then the model to be trained is trained based on the cross-entropy loss to obtain the key point extraction model.

[0070] In the embodiment of the present application, based on different key point extraction requirements, preset thresholds of different repeatability can be set to screen out key points of different repeatability, so as to complete the training of key point extraction models for different extraction effects. The model is trained by the repeatability of key points under different visual transformations, so as to improve the consistency of the model in extracting key points under different visual conditions (viewing angles, illumination), and to improve the accuracy of key point extraction.

[0071] Phase 2:

[0072] It is understandable that the position of the same key point under different viewing angles should be consistent. Therefore, images of multiple viewing angles containing the same key point can be collected, and the model can be optimized by predicting whether the positions of the same key point on images of different viewing angles are consistent. That is, optionally, in some embodiments of the present application, the key point extraction model is obtained by the following steps:

[0073] Acquire at least one set of second sample image sets, where the second sample image set includes two sample scene images, where the two sample scene images are scene images of the same scene from different perspectives;

[0074] For any second sample image set, extract key point information from each image in the second sample image set by using the first model to be trained to obtain at least one sample key point information corresponding to each image in the second sample image set;

[0075] For any sample scene image in the second sample image set, projecting a sample key point corresponding to any sample key point information of the sample scene image onto another sample scene image in the second sample image set to obtain projected key point information of the sample key point on the other sample scene image;

[0076] According to the sample key point information of each sample key point in the other sample scene image and the projection key point information corresponding to each sample key point, the first model to be trained is trained to obtain a key point extraction model.

[0077] The two sample scene images in the second sample image set are scene images of the same scene from different perspectives, and the two sample scene images include at least one identical key point.

[0078] It can be understood that after a model with better effects extracts two sample scene images, the information of the same key point in the two sample scene images should be consistent. Therefore, the embodiment of the present application mainly predicts the key points in the two sample scene images respectively based on the model to be trained, and then optimizes the model by comparing the differences between the two key points.

[0079] Specifically: Take two sample scene images, image A and image B, as examples, where the relative position, posture and depth map corresponding to image A and image B are known information, extract the key point 1 in image A through the model to be trained to obtain its position Pa, and extract the key point 1 in image B to obtain its position Pb, where Pa and Pb are the same key point 1 in image A and image B. Through the relative position, posture and depth map, the key point 1 in image A is projected into image B to obtain position Pab. It can be understood that when the extraction effect of the model to be trained is good, position Pab and position Pb should overlap. Therefore, in the embodiment of the present application, in the training stage, the model to be trained is trained based on the distance between position Pab and position Pb to obtain a key point extraction model to shorten the distance between position Pab and position Pb.

[0080] Among them, in the embodiment of the present application, the first model to be trained is a model to be trained based on the distance between position Pab and position Pb. Accordingly, the position Pab and position Pb corresponding to each key point can constitute a matching point pair, and the model can be trained through each matching point pair corresponding to each key point.

[0081] Optionally, in an embodiment of the present application, in order to constrain the key points to the corner positions where the texture gradient is obvious, rather than the similar texture area, a peak loss can be added in the training to constrain the response map of the key points. Specifically, in some embodiments of the present application, the step of "training the first model to be trained to obtain a key point extraction model according to the sample key point information of each sample key point in the other sample scene image and the projection key point information corresponding to each sample key point" includes:

[0082] Determine sample key point distance information according to sample key point information of the sample key point in the other sample scene image and projected key point information, and construct first loss information according to the sample key point distance information;

[0083] For each sample key point information of each sample scene image in the second sample image set, determine a sample region image and a sample region response map corresponding to the sample region image from the sample scene image corresponding to the sample key point information, wherein the sample region response map includes a probability score of each pixel point in the sample region image belonging to a key point;

[0084] A maximum probability value with the largest probability score is selected from the sample area response map, and the probability score of each pixel in the sample area image is normalized according to the maximum probability value to obtain a normalized probability score corresponding to each pixel;

[0085] Calculate the target key point coordinate information according to the position information and normalized probability score of each pixel in the sample area response map;

[0086] The second loss information is obtained by calculating the distance information between each pixel point and the target key point coordinate information in the sample area response map and the normalized probability score of each pixel point;

[0087] The first model to be trained is trained according to the first loss information and the second loss information corresponding to the key point information of each sample to obtain a key point extraction model.

[0088] It can be understood that the first loss information is the distance information corresponding to the matching point pair, and the second loss information is for the peak loss.

[0089] It should be noted that the sample area image is a small area image in the sample scene image, such as a pixel block of size NxN, and the sample area response map is a response map corresponding to the sample area image. The sample area response map contains the probability values ​​corresponding to each pixel point in the sample area image, and the probability value is the probability value of the pixel point being a key point. It can be understood that the larger the probability value, the higher the probability that the pixel point is a key point.

[0090] Among them, the normalized probability score is obtained by normalizing the maximum probability value in the sample area response map, so that each normalized probability score is expressed as a probability distribution, which clearly indicates the relative importance of each pixel in the entire response map, that is, reflects the probability of each pixel being a key point. Therefore, the probability of each pixel being a key point can be more accurately expressed based on the normalized probability score.

[0091] Among them, since the purpose of the embodiment of the present application is to constrain the position of the key point, the key points in the sample area image block must meet the following requirements: first, the farther the point is from the key point p, the smaller its response map score is. Second, the response value of the key point position is large enough relative to the pixels around it. Training is performed as the target.

[0092] Solving the key points of the sample area image (predicting the key point coordinates) is transformed into solving the expected probability distribution problem, that is, the key point coordinates of the sample area image reflect the average value of the random variable. In this case, the coordinates of the sample area response map are used as random variables, and their probability distribution is determined by the normalized probability score. By calculating the expectation (expectation under the probability distribution), the probability and coordinate information of each position in the sample area response map can be comprehensively considered to obtain a more accurate prediction of the key point coordinates. The specific calculation method is to multiply the coordinates of each position in the sample area response map (each pixel corresponds to a position coordinate) by the normalized probability score of the position, and then sum the products of all positions.

[0093] Furthermore, after obtaining the key point coordinates of the sample area image (i.e., the target key point coordinate information, which is also the predicted key point coordinates mentioned above), the loss can be constructed based on the distance between each pixel point in the sample area image and the predicted key point coordinates and the normalized probability score of each pixel point.

[0094] Specifically, for a pixel block of size NxN, the corresponding response map is ; The center point coordinates of the NxN pixel block are [u, v], and the maximum value in its response graph is ; For NxN pixel blocks The score value at the position (i.e., the corresponding value in the sample area response map) is calculated as follows: .

[0095] but Represents pixel is the probability of the key point (i.e., the normalized probability score). Then the position of the key point should be the "center of mass" of the NxN pixel block, which is expressed as:

[0096] .

[0097] The peak loss (corresponding to: the second loss information) is defined as:

[0098] .

[0099] The peak loss represents the distance between each pixel and the key point p multiplied by the probability value of the pixel being a key point (i.e., the normalized probability score). In combination with the peak loss formula, there are two goals to be achieved in order to reduce the peak loss. First, the farther the point is from the key point p, the smaller its response map score. Second, the response value of the key point position is large enough relative to the pixels around it. In other words, the response value around the key point can be made to have a peak distribution, and the similarity between the key point and the surrounding pixels is not high (corner points, not similar texture areas).

[0100] In summary, in the embodiments of the present application, the key point extraction model obtained by training can be divided into two stages, the first stage is based on the key point repeatability training, and the second stage is based on the distance between the matching point pairs and the peak loss training. In some feasible embodiments of the present application, the first stage and the second stage may not have a strict order, for example, the first stage is in front and the second stage is in the back, the first stage may also be in the back and the second stage is in the front, and in some cases, the first stage and the second stage may also be performed simultaneously.

[0101] Among them, in the embodiments of the present application, the first stage is first and the second stage is later as an example. For example, when constructing the peak loss in the second stage, the sample area response map involved is generated by the model obtained after the first stage training. The response map output by the model after the second stage training is constrained by the design of the peak loss, so that the extraction effect of the model is better.

[0102] It can be understood that the key point extraction model trained in the embodiment of the present application can extract key point information from the target image, and based on this purpose, the structure of the key point extraction model can be flexibly designed, such as Figure 4 As shown, Figure 4 : is a structural diagram of a key point extraction model provided by an embodiment of the present application. Specifically, the key point extraction model specifically includes a recombination unit, a first convolution unit, a second convolution unit, a third convolution unit and an activation unit. The input end of the recombination unit is configured to receive model input information of the key point extraction model, the output end of the recombination unit is connected to the input end of the first convolution unit, the output end of the first convolution unit is connected to the input end of the second convolution unit, the output end of the second convolution unit is connected to the input end of the third convolution unit, the output end of the third convolution unit is connected to the input end of the activation unit, and the output end of the activation unit is configured to output the model output information of the key point extraction model;

[0103] Wherein, the first convolution unit includes at least one first convolution layer, the second convolution unit includes at least one second convolution layer, and the third convolution unit includes at least one third convolution layer;

[0104] Among them, at least one dimension of the activation unit is used to represent a flag bit, and the flag bit represents at least one of the target image containing key point information and not containing key point information. Among them, other dimensions in the activation unit except the dimension of the flag bit are used to determine the position information of the key point through the probability value corresponding to each pixel point, and the position information is output to obtain the key point information.

[0105] For details, please refer to Figure 4 As shown, the first convolution layer includes a 3×3 convolution layer, the second convolution layer includes a 1×1 convolution layer, and the third convolution layer includes a 1×1 convolution layer.

[0106] Specifically, take the trained key point extraction model to process the H×W target image as an example. Specifically, the grayscale image of the H×W target image is input into the key point extraction model, and the grayscale image is expanded into a 64×H / 8×W / 8 tensor through the reorganization unit. After being processed by 8 3×3 convolutional layers of the first convolutional unit, 16 1×1 convolutional layers of the second convolutional unit, and 65 1×1 convolutional layers of the third convolutional unit, a 65×H / 8×W / 8 tensor is obtained, where the last bit of the 65 dimension is a flag bit, indicating whether there is a key point in the 8×8 grid. The remaining 64 channel dimensions Softmax operation can normalize the 8×8 grid, where the position of the maximum value is the position of the key point (that is, the key point information). The 8×8 grid is a square pixel block of 8×8 pixels. For example, a 320×240 image can be divided into 40×30 8×8 pixel blocks.

[0107] It can be understood that the descriptor of a feature point is a vector of a specific length formed by encoding the attributes of the local area around it. By calculating the distance between the feature vectors, the similarity between feature points can be determined, thereby completing the matching between multiple feature points on different pictures. Among them, the key points (such as corner points) in the same physical space will change their values ​​on the picture pixels significantly as the lighting and observation angle distance change; while the corner points in different physical spaces may have similar values ​​on the picture pixels due to the similarity in appearance. The difficulty of descriptor extraction lies in how to design the calculation method of the descriptor so that the same feature point has a close distance between its descriptor feature vectors when the lighting and observation angle distance change, while the distance between the descriptor feature vectors of different feature points is far.

[0108] To this end, the embodiment of the present application trains the second to-be-trained model to be trained with similar situations of the descriptors of the feature points or the descriptors corresponding to the key points. Specifically, in some embodiments of the present application, the descriptor extraction model is obtained by the following steps:

[0109] Acquire at least one set of third sample image sets, where the third sample image set includes two sample scene images, where the two sample scene images are scene images of the same scene from different perspectives;

[0110] For any third sample image set, extract descriptor information of each image in the third sample image set by using the second to-be-trained model to obtain sample descriptor information corresponding to each sample key point information corresponding to each image;

[0111] For each sample descriptor information of each sample scene image in the third sample image set, calculating the similarity between the sample descriptor information and each sample descriptor information corresponding to another sample scene image in the third sample image set to obtain description similarity information;

[0112] Generate target loss information according to each description similarity information corresponding to each sample descriptor information in each sample scene image in the third sample image set;

[0113] The second model to be trained is trained according to the target loss information to obtain a descriptor extraction model.

[0114] It can be understood that the description similarity information represents the similarity between descriptors. Building a loss based on the similarity to train the model helps the model train according to the characteristics that the distance between the descriptor feature vectors of the same feature point is close when the illumination and observation angle change, while the distance between the descriptor feature vectors of different feature points is far. This can improve the accuracy of the descriptor extraction model after training, that is, improve the accuracy of the descriptors extracted by the descriptor extraction model after training.

[0115] Among them, in order to achieve the above purpose, points with matching descriptor information for the same key point can be screened out from the similarity information to construct the loss. For example, for each sample descriptor information, the matching descriptor information for the same key point is screened out based on the description similarity information between the sample descriptor information and each sample descriptor information in another sample scene image, and the model is trained based on the close distance of the descriptor information. Specifically, in some embodiments of the present application, the step of "generating target loss information according to each description similarity information corresponding to each sample descriptor information in each sample scene image in the third sample image set" includes:

[0116] For each sample descriptor information in each sample scene image in the third sample image set, normalizing each description similarity information corresponding to the sample descriptor information to obtain normalized similarity set information;

[0117] Determining sample matching descriptor information corresponding to the sample descriptor information from another sample scene image of the third sample image set, and extracting normalized similarity information corresponding to the sample matching descriptor information from the normalized similarity set information;

[0118] Perform negative logarithm processing on the normalized similarity information to construct sub-loss information for describing the sub-information of the sample;

[0119] The target loss information is obtained by calculating the average value of the sub-loss information of each sample descriptor sub-information of each sample scene image in the third sample image set.

[0120] It should be noted that the sample matching descriptor information is the descriptor information that matches the sample descriptor information, for example, the key point in image A The corresponding sample descriptor information, the corresponding sample matching descriptor information is the key point in image B The corresponding key points Sample descriptor information of With key points It is the representation of the same key point on different images. Also marked as In this embodiment of the present application, the key points in picture A The corresponding key points , can be based on key points The similarity information of the descriptor information of each key point in image B is used to determine the similarity. The corresponding key points In order to improve the accuracy of the key points, the embodiment of the present application adopts The similarity between the descriptor information of each key point in image B and the key point in image B is normalized, and the key points in image B are comprehensively identified based on the normalized results. The best matching keypoint .

[0121] For example, the third sample image set includes image A and image B, and image A contains key points Take this as an example to explain, specifically:

[0122] Extract key points in image A through the second model to be trained Descriptor , and extract the descriptors of all key points in image B ;

[0123] Calculate the key points in image A Descriptor , and the descriptors of all key points in image B The dot product between them gives a similar graph = ( ,D B );

[0124] Normalize the similarity graph to get the normalized similarity graph:

[0125] .

[0126] The corresponding NRE loss function can calculate the cross entropy, specifically:

[0127] .

[0128] in, It is the set of all key point information in image B.

[0129] in, is the key point in image A The true probability that all key points in image B are matching points is, where, is the key point in image B.

[0130] Among them, due to the matching point ( ) is 1, while the probability of the remaining key points is 0, so only the cross entropy loss of the true matching points is calculated. Obviously, only the matching points , the rest are 0, so the key point The loss function can be simplified to the final single form. Specifically: .

[0131] Finally, all and The corresponding loss functions are added together and averaged to get the final NRE loss function, that is, the target loss information.

[0132] It can be understood that the loss design with a true probability of 1 helps to achieve the purpose of keeping the distance between the descriptor feature vectors of the same feature point close when the illumination and observation angle change, but it also implies the normalization. The probability sum of each key point in image B is limited to 1, so as to achieve are non-matching points (except The probability of key points other than the matching points is restricted, which reduces the probability scores of these non-matching points. Therefore, based on normalization, the purpose of making the distance between the feature vectors of the descriptor information of different feature points far apart is actually achieved.

[0133] It can be understood that the descriptor extraction model trained in the embodiment of the present application can extract the descriptor information corresponding to the key point information from the target image, and based on this purpose, the structure of the descriptor extraction model can be flexibly designed, such as Figure 5 As shown, Figure 5: is a structural diagram of a descriptor extraction model provided by an embodiment of the present application. Specifically, the descriptor extraction model includes a fourth convolution unit, a fifth convolution unit, a sixth convolution unit, a seventh convolution unit, a first sampling unit, a second sampling unit, an eighth convolution unit, a fusion unit, a third sampling unit and a ninth convolution unit. The input end of the fourth convolution unit is configured to receive model input information of the descriptor extraction model, the output end of the fourth convolution unit is respectively connected to the input end of the fifth convolution unit, the input end of the sixth convolution unit and the input end of the seventh convolution unit, the output end of the fifth convolution unit is connected to the input end of the fusion unit, the output end of the sixth convolution unit is connected to the input end of the first sampling unit, the output end of the first sampling unit is connected to the input end of the fusion unit, the output end of the seventh convolution unit is connected to the input end of the second sampling unit, the output end of the second sampling unit is connected to the input end of the eighth convolution unit, the output end of the eighth convolution unit is connected to the input end of the fusion unit, the output end of the fusion unit is connected to the input end of the third sampling unit, the output end of the third sampling unit is connected to the input end of the ninth convolution unit, and the output end of the ninth convolution unit is configured to output the model output information of the descriptor extraction model.

[0134] Specifically, in the embodiment of the present application, the fourth convolution unit includes a 3×3 convolution layer and a 1×1 convolution layer, the fifth convolution unit, the sixth convolution unit, and the seventh convolution unit include a 3×3 convolution layer, and the eighth convolution unit includes a 1×1 convolution layer.

[0135] For example, see Figure 5 As shown, taking the trained descriptor extraction model to process the H×W target image as an example, specifically, the grayscale image of the H×W target image is input into the descriptor extraction model, and the grayscale image is processed through the skip connection and the fourth convolution unit to obtain a 16×H / 4×W / 4 tensor.

[0136] In order to fuse multi-scale information, three groups of feature maps with different resolutions are designed, that is, the 16×H / 4×W / 4 tensor is processed by the fifth convolution unit to obtain a 48×H / 8×W / 8 tensor, the 16×H / 4×W / 4 tensor is processed by the sixth convolution unit to obtain a 48×H / 16×W / 16 tensor, and the 16×H / 4×W / 4 tensor is processed by the seventh convolution unit to obtain a 96×H / 32×W / 32 tensor. Furthermore, in an embodiment of the present application, the 48×H / 16×W / 16 tensor obtained by the sixth convolution unit is upsampled to obtain a 48×H / 16×W / 16 tensor, and the 96×H / 32×W / 32 tensor obtained by the seventh convolution unit is upsampled and convolved with 1×1 to obtain a 48×H / 16×W / 16 tensor.

[0137] The 48×H / 16×W / 16 tensors output by the fifth convolution unit, the sixth convolution unit, and the seventh convolution unit after final processing are fused through a fusion unit to obtain an initial feature map of 48×H / 8×W / 8. The initial feature map corresponds to the description information of the target image in the embodiment of the present application.

[0138] Then, according to the key point information of the target image, a plurality of descriptor information corresponding to the key point information is sampled from the initial feature map by a third sampling unit, such as Figure 5 The M description offset information of the third sampling unit in the image (wherein the description offset information is obtained after the descriptor extraction model is trained) is used, and M descriptor information of the key points is extracted based on the M description offset information. Subsequently, the descriptor information finally corresponding to the key point information can be obtained by convolution based on the M descriptor information. For example, the key point extraction model is used to extract N key point information of the target image, and a 48×N×M tensor is extracted from the initial feature map based on the M description offset information. The 48×N×M tensor is convolved with the ninth convolution unit to obtain 48×N descriptor information.

[0139] For example, for a single key point distance, if the key point (i, j), the description offset information offset = (2, 5); then the descriptor (48-dimensional vector) at the position (i+2, j+5) will be taken, and M description offset information will be used to obtain M 48-dimensional vectors. Through convolution, a final 48-dimensional vector can be obtained, which is the descriptor information corresponding to the key point (i, j).

[0140] Among them, in the embodiment of the present application, the design of the third sampling unit (description offset information) enables the descriptor to adaptively fuse information at different positions in the image, enhance the descriptor's description ability, and improve the final matching effect. By fusing feature maps of different resolutions to obtain the description information corresponding to the target image, the semantic information of different receptive field sizes can be captured, and attention can be paid to multi-level information and semantic information. Multi-layer feature fusion and dynamic sampling features are used to improve the accuracy of the descriptor information corresponding to the key point information.

[0141] It is understandable that in the embodiments of the present application, in order to improve the effect of the model, the model can also be trained based on the feature points or key points of the application. For example, some applications only need 50 feature points, while some applications hope to have 500 feature points. Therefore, according to the needs of the application, the format of the feature points is set during training to make the model expression more suitable for the application.

[0142] In summary, the embodiments of the present application obtain the key point information and description information of the target image by extracting the key point information and the description information, and determining the corresponding descriptor information from the description information based on the key point information, thereby obtaining the feature point information including the key point information and the descriptor information corresponding to the key point information.

[0143] Among them, by training the key point extraction model and the descriptor extraction model separately, it is helpful to use the key point extraction model alone to extract the key point information of the target image, and realize the extraction of key point information, and it is also helpful to use the descriptor extraction model alone to extract the description information of the target image. Such a solution of processing key point information and description information separately is helpful for processing downstream tasks of key point information analysis or description information analysis, that is, it is easier to meet the needs of downstream tasks. Among them, training the key point extraction model and the descriptor extraction model separately also helps to improve the training efficiency of the key point extraction model and the descriptor extraction model.

[0144] Among them, when the key point extraction model and the descriptor extraction model are trained separately, since the key points usually do not require a large receptive field, the key point extraction model can be designed as a very shallow network, and the channel dimension of the descriptor extraction model is also relatively low. It mainly fuses the information of each level through a multi-layer network, and fuses the effective information area through offset (description offset information), so as to enhance the network's expression ability under limited network complexity. At the same time, the shallow network corresponding to the key point extraction model and the relatively low channel dimension of the descriptor extraction model make the key point extraction model and the descriptor extraction model lightweight, occupying a small amount of computing resources on a resource-constrained platform, while achieving excellent algorithm effects, that is, low power consumption and longer device battery life, such as minimizing battery life issues when applied to augmented reality devices.

[0145] In order to better implement the information extraction method of the present application, the present application also provides an information extraction device based on the above information extraction method. The meanings of the terms are the same as those in the above information extraction method, and the specific implementation details can refer to the description in the method embodiment.

[0146] See also Figure 6 , Figure 6 : is a schematic diagram of the structure of the information extraction device provided in an embodiment of the present application, wherein the information extraction device can be specifically as follows:

[0147] An acquisition module 201 is used to acquire a target image to be processed;

[0148] An extraction module 202, used to extract key point information of a target image through a key point extraction model, and to extract description information of the target image through a descriptor extraction model;

[0149] A determination module 203, configured to determine the descriptor information corresponding to the key point information from the description information according to the description offset information corresponding to the descriptor extraction model;

[0150] A generating module 204, configured to generate image feature point information of a target image according to the key point information and the descriptor information corresponding to the key point information;

[0151] Among them, the key point extraction model and the descriptor extraction model are trained separately.

[0152] Optionally, in some embodiments of the present application, the key point extraction model is obtained by the following steps:

[0153] Acquire at least one set of first sample image sets, the first sample image set comprising a sample original image and at least one sample transformed image obtained by performing a visual transformation on the sample original image, wherein the visual transformation comprises at least one of an illumination transformation and a perspective transformation;

[0154] For any first sample image set, extract key point information from each image in the first sample image set to obtain at least one sample key point information corresponding to each image;

[0155] For any sample key point information, obtaining extraction ratio information according to the number of pictures from which the sample key point information is extracted in the first sample image set and the total number of pictures corresponding to the first sample image set;

[0156] The key point extraction model is trained according to the key point information of each sample and the extraction ratio information corresponding to the key point information of each sample.

[0157] Optionally, in some embodiments of the present application, the key point extraction model is obtained by the following steps:

[0158] Acquire at least one set of second sample image sets, where the second sample image set includes two sample scene images, where the two sample scene images are scene images of the same scene from different perspectives;

[0159] For any second sample image set, extract key point information from each image in the second sample image set by using the first model to be trained to obtain at least one sample key point information corresponding to each image in the second sample image set;

[0160] For any sample scene image in the second sample image set, projecting a sample key point corresponding to any sample key point information of the sample scene image onto another sample scene image in the second sample image set to obtain projected key point information of the sample key point on the other sample scene image;

[0161] According to the sample key point information of each sample key point in the other sample scene image and the projection key point information corresponding to each sample key point, the first model to be trained is trained to obtain a key point extraction model.

[0162] Optionally, in some embodiments of the present application, according to the sample key point information of each sample key point in the other sample scene image and the projection key point information corresponding to each sample key point, the first to-be-trained model is trained to obtain a key point extraction model, including:

[0163] Determine sample key point distance information according to sample key point information of the sample key point in the other sample scene image and projected key point information, and construct first loss information according to the sample key point distance information;

[0164] For each sample key point information of each sample scene image in the second sample image set, determine a sample region image and a sample region response map corresponding to the sample region image from the sample scene image corresponding to the sample key point information, wherein the sample region response map includes a probability score of each pixel point in the sample region image belonging to a key point;

[0165] A maximum probability value with the largest probability score is selected from the sample area response map, and the probability score of each pixel in the sample area image is normalized according to the maximum probability value to obtain a normalized probability score corresponding to each pixel;

[0166] Calculate the target key point coordinate information according to the position information and normalized probability score of each pixel in the sample area response map;

[0167] The second loss information is obtained by calculating the distance information between each pixel point and the target key point coordinate information in the sample area response map and the normalized probability score of each pixel point;

[0168] The first model to be trained is trained according to the first loss information and the second loss information corresponding to the key point information of each sample to obtain a key point extraction model.

[0169] Optionally, in some embodiments of the present application, the key point extraction model includes a recombination unit, a first convolution unit, a second convolution unit, a third convolution unit and an activation unit, the input end of the recombination unit is configured to receive model input information of the key point extraction model, the output end of the recombination unit is connected to the input end of the first convolution unit, the output end of the first convolution unit is connected to the input end of the second convolution unit, the output end of the second convolution unit is connected to the input end of the third convolution unit, the output end of the third convolution unit is connected to the input end of the activation unit, and the output end of the activation unit is configured to output the model output information of the key point extraction model;

[0170] Wherein, the first convolution unit includes at least one first convolution layer, the second convolution unit includes at least one second convolution layer, and the third convolution unit includes at least one third convolution layer;

[0171] Among them, at least one dimension of the activation unit is used to represent a flag bit, and the flag bit represents at least one of the target image containing key point information and not containing key point information. Among them, other dimensions in the activation unit except the dimension of the flag bit are used to determine the position information of the key point through the probability value corresponding to each pixel point, and the position information is output to obtain the key point information.

[0172] Optionally, in some embodiments of the present application, the descriptor extraction model is obtained by the following steps:

[0173] Acquire at least one set of third sample image sets, where the third sample image set includes two sample scene images, where the two sample scene images are scene images of the same scene from different perspectives;

[0174] For any third sample image set, extract descriptor information of each image in the third sample image set by using the second to-be-trained model to obtain sample descriptor information corresponding to each sample key point information corresponding to each image;

[0175] For each sample descriptor information of each sample scene image in the third sample image set, calculating the similarity between the sample descriptor information and each sample descriptor information corresponding to another sample scene image in the third sample image set to obtain description similarity information;

[0176] Generate target loss information according to each description similarity information corresponding to each sample descriptor information in each sample scene image in the third sample image set;

[0177] The second model to be trained is trained according to the target loss information to obtain a descriptor extraction model.

[0178] Optionally, in some embodiments of the present application, generating target loss information according to each description similarity information respectively corresponding to each sample descriptor information in each sample scene image in the third sample image set includes:

[0179] For each sample descriptor information in each sample scene image in the third sample image set, normalizing each description similarity information corresponding to the sample descriptor information to obtain normalized similarity set information;

[0180] Determining sample matching descriptor information corresponding to the sample descriptor information from another sample scene image of the third sample image set, and extracting normalized similarity information corresponding to the sample matching descriptor information from the normalized similarity set information;

[0181] Perform negative logarithm processing on the normalized similarity information to construct sub-loss information for describing the sub-information of the sample;

[0182] The target loss information is obtained by calculating the average value of the sub-loss information of each sample descriptor sub-information of each sample scene image in the third sample image set.

[0183] Optionally, in some embodiments of the present application, the descriptor extraction model includes a fourth convolution unit, a fifth convolution unit, a sixth convolution unit, a seventh convolution unit, a first sampling unit, a second sampling unit, an eighth convolution unit, a fusion unit, a third sampling unit and a ninth convolution unit, the input end of the fourth convolution unit is configured to receive model input information of the descriptor extraction model, the output end of the fourth convolution unit is respectively connected to the input end of the fifth convolution unit, the input end of the sixth convolution unit and the input end of the seventh convolution unit, the output end of the fifth convolution unit is connected to the input end of the fusion unit, the output end of the sixth convolution unit is connected to the input end of the first sampling unit, the output end of the first sampling unit is connected to the input end of the fusion unit, the output end of the seventh convolution unit is connected to the input end of the second sampling unit, the output end of the second sampling unit is connected to the input end of the eighth convolution unit, the output end of the eighth convolution unit is connected to the input end of the fusion unit, the output end of the fusion unit is connected to the input end of the third sampling unit, the output end of the third sampling unit is connected to the input end of the ninth convolution unit, and the output end of the ninth convolution unit is configured to output model output information of the descriptor extraction model.

[0184] In the embodiment of the present application, the acquisition module 201 first acquires the target image to be processed, and then the extraction module 202 extracts the key point information of the target image through the key point extraction model, and extracts the description information of the target image through the descriptor extraction model. Subsequently, the determination module 203 determines the descriptor information corresponding to the key point information from the description information according to the description offset information corresponding to the descriptor extraction model. Then, the generation module 204 generates the image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information, wherein the key point extraction model and the descriptor extraction model are trained separately.

[0185] Among them, the embodiment of the present application obtains the key point information and description information of the target image by extracting, and determines the corresponding descriptor information from the description information based on the key point information, thereby achieving the acquisition of feature point information containing the key point information and the descriptor information corresponding to the key point information.

[0186] Among them, by training the key point extraction model and the descriptor extraction model separately, it is helpful to use the key point extraction model alone to extract the key point information of the target image, and realize the extraction of key point information, and it is also helpful to use the descriptor extraction model alone to extract the description information of the target image. Such a solution of processing key point information and description information separately is helpful for processing downstream tasks of key point information analysis or description information analysis, that is, it is easier to meet the needs of downstream tasks. Among them, training the key point extraction model and the descriptor extraction model separately also helps to improve the training efficiency of the key point extraction model and the descriptor extraction model.

[0187] In addition, the present application also provides an electronic device, such as Figure 7 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0188] The electronic device may include one or more processors 301 of processing cores, one or more computer-readable storage media memories 302, a power supply 303, an input unit 304, a camera 305 and other components. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0189] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0190] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0191] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0192] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0193] The camera 305 is used to capture images.

[0194] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby implementing the steps in any one of the information extraction methods provided in the embodiments of the present application.

[0195] The embodiment of the present application obtains a target image to be processed, extracts key point information of the target image through a key point extraction model, and extracts description information of the target image through a descriptor extraction model, determines descriptor information corresponding to the key point information from the description information according to description offset information corresponding to the descriptor extraction model, and generates image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information. The key point extraction model and the descriptor extraction model are trained separately.

[0196] The key point information and description information of the target image are extracted, and the corresponding descriptor information is determined from the description information according to the key point information, so as to obtain the feature point information including the key point information and the descriptor information corresponding to the key point information.

[0197] Among them, by training the key point extraction model and the descriptor extraction model separately, it is helpful to use the key point extraction model alone to extract the key point information of the target image, and realize the extraction of key point information, and it is also helpful to use the descriptor extraction model alone to extract the description information of the target image. Such a solution of processing key point information and description information separately is helpful for processing downstream tasks of key point information analysis or description information analysis, that is, it is easier to meet the needs of downstream tasks. Among them, training the key point extraction model and the descriptor extraction model separately also helps to improve the training efficiency of the key point extraction model and the descriptor extraction model.

[0198] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0199] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0200] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any one of the information extraction methods provided in the present application.

[0201] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0202] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0203] Since the instructions stored in the computer-readable storage medium can execute the steps in any information extraction method provided in the present application, the beneficial effects that can be achieved by any information extraction method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0204] The above is a detailed introduction to an information extraction method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An information extraction method, characterized in that: The method comprises: Obtaining a target image to be processed; Extracting key point information of the target image through a key point extraction model, and extracting description information of the target image through a descriptor extraction model; Determining the descriptor information corresponding to the key point information from the description information according to the description offset information corresponding to the descriptor extraction model; Generate image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information; Wherein, the key point extraction model and the descriptor extraction model are trained separately; The key point extraction model is obtained through the following steps: Acquire at least one set of first sample image sets, wherein the first sample image set includes a sample original image and at least one sample transformed image obtained by performing a visual transformation on the sample original image, wherein the visual transformation includes at least one of an illumination transformation and a perspective transformation; For any of the first sample image sets, extract key point information from each image in the first sample image set to obtain at least one sample key point information corresponding to each image; For any sample key point information, obtaining extraction ratio information according to the number of pictures from which the sample key point information is extracted in the first sample image set and the total number of pictures corresponding to the first sample image set; The key point extraction model is trained according to the key point information of each sample and the extraction ratio information corresponding to the key point information of each sample.

2. The information extraction method according to claim 1, characterized in that: The key point extraction model is obtained through the following steps: Acquire at least one set of second sample image sets, where the second sample image set includes two sample scene images, where the two sample scene images are scene images of the same scene from different perspectives; For any of the second sample image sets, extract key point information of each image in the second sample image set by using the first to-be-trained model to obtain at least one sample key point information corresponding to each image in the second sample image set; For any sample scene image in the second sample image set, projecting a sample key point corresponding to any sample key point information of the sample scene image onto another sample scene image in the second sample image set to obtain projected key point information of the sample key point on the other sample scene image; According to the sample key point information of each sample key point in the other sample scene image and the projection key point information corresponding to each sample key point, the first model to be trained is trained to obtain a key point extraction model.

3. The information extraction method according to claim 2, characterized in that: The step of training the first to-be-trained model to obtain a key point extraction model according to the sample key point information of each sample key point in the other sample scene image and the projection key point information corresponding to each sample key point includes: For each sample key point, determine the sample key point distance information according to the sample key point information of the sample key point in the other sample scene image and the projected key point information of the sample key point in the other sample scene image, and construct the first loss information according to the sample key point distance information; For each sample key point information of each sample scene image in the second sample image set, determine a sample region image and a sample region response map corresponding to the sample region image from the sample scene image corresponding to the sample key point information, wherein the sample region response map includes a probability score of each pixel point in the sample region image belonging to a key point; A maximum probability value with the largest probability score is selected from the sample area response map, and the probability score of each pixel in the sample area image is normalized according to the maximum probability value to obtain a normalized probability score corresponding to each pixel; Calculate the target key point coordinate information according to the position information and normalized probability score of each pixel in the sample area response map; The second loss information is obtained by calculating the distance information between each pixel point and the target key point coordinate information in the sample area response map and the normalized probability score of each pixel point; The first model to be trained is trained according to the first loss information and the second loss information corresponding to the key point information of each sample to obtain a key point extraction model.

4. The information extraction method according to claim 1, characterized in that: The key point extraction model includes a recombination unit, a first convolution unit, a second convolution unit, a third convolution unit and an activation unit, the input end of the recombination unit is configured to receive model input information of the key point extraction model, the output end of the recombination unit is connected to the input end of the first convolution unit, the output end of the first convolution unit is connected to the input end of the second convolution unit, the output end of the second convolution unit is connected to the input end of the third convolution unit, the output end of the third convolution unit is connected to the input end of the activation unit, and the output end of the activation unit is configured to output the model output information of the key point extraction model; The first convolution unit includes at least one first convolution layer, the second convolution unit includes at least one second convolution layer, and the third convolution unit includes at least one third convolution layer; At least one dimension of the activation unit is used to represent a flag bit, and the flag bit represents at least one of whether the target image contains key point information and does not contain key point information. For other dimensions in the activation unit except the dimension of the flag bit, the position information of the key point is determined by the probability value corresponding to each pixel point, and the position information is output to obtain the key point information.

5. The information extraction method according to claim 1, characterized in that: The descriptor extraction model is obtained by the following steps: Acquire at least one set of third sample image sets, wherein the third sample image set includes two sample scene images, and the two sample scene images are scene images of the same scene from different perspectives; For any of the third sample image sets, extract descriptor information of each image in the third sample image set by using the second to-be-trained model to obtain sample descriptor information corresponding to each sample key point information corresponding to each image; For each sample descriptor information of each sample scene image in the third sample image set, calculating the similarity between the sample descriptor information and each sample descriptor information corresponding to another sample scene image in the third sample image set to obtain description similarity information; Generate target loss information according to each description similarity information corresponding to each sample descriptor information in each sample scene image in the third sample image set; The second model to be trained is trained according to the target loss information to obtain a descriptor extraction model.

6. The information extraction method according to claim 5, characterized in that: The generating target loss information according to each description similarity information respectively corresponding to each sample descriptor information in each sample scene image in the third sample image set comprises: For each sample descriptor information in each sample scene image in the third sample image set, normalizing each description similarity information corresponding to the sample descriptor information to obtain normalized similarity set information; Determining sample matching descriptor information corresponding to the sample descriptor information from another sample scene image of the third sample image set, and extracting normalized similarity information corresponding to the sample matching descriptor information from the normalized similarity set information; Performing negative logarithm processing on the normalized similarity information to construct sub-loss information for describing the sub-information of the sample; The target loss information is obtained by calculating the average value of the sub-loss information of each sample descriptor sub-information of each sample scene image in the third sample image set.

7. The information extraction method according to claim 1, characterized in that: The descriptor extraction model includes a fourth convolution unit, a fifth convolution unit, a sixth convolution unit, a seventh convolution unit, a first sampling unit, a second sampling unit, an eighth convolution unit, a fusion unit, a third sampling unit and a ninth convolution unit, the input end of the fourth convolution unit is configured to receive model input information of the descriptor extraction model, the output end of the fourth convolution unit is respectively connected to the input end of the fifth convolution unit, the input end of the sixth convolution unit and the input end of the seventh convolution unit, the output end of the fifth convolution unit is connected to the input end of the fusion unit, and the output end of the sixth convolution unit is connected to the input end of the fusion unit. The input end of the first sampling unit is connected, the output end of the first sampling unit is connected to the input end of the fusion unit, the output end of the seventh convolution unit is connected to the input end of the second sampling unit, the output end of the second sampling unit is connected to the input end of the eighth convolution unit, the output end of the eighth convolution unit is connected to the input end of the fusion unit, the output end of the fusion unit is connected to the input end of the third sampling unit, the output end of the third sampling unit is connected to the input end of the ninth convolution unit, and the output end of the ninth convolution unit is configured to output the model output information of the descriptor extraction model.

8. An information extraction device, characterized in that: The device comprises: An acquisition module, used for acquiring a target image to be processed; An extraction module, used to extract key point information of the target image through a key point extraction model, and to extract description information of the target image through a descriptor extraction model; A determination module, configured to determine the descriptor information corresponding to the key point information from the description information according to the description offset information corresponding to the descriptor extraction model; A generating module, used for generating image feature point information of the target image according to the key point information and the descriptor information corresponding to the key point information; Wherein, the key point extraction model and the descriptor extraction model are trained separately; The key point extraction model is obtained through the following steps: Acquire at least one set of first sample image sets, wherein the first sample image set includes a sample original image and at least one sample transformed image obtained by performing a visual transformation on the sample original image, wherein the visual transformation includes at least one of an illumination transformation and a perspective transformation; For any of the first sample image sets, extract key point information from each image in the first sample image set to obtain at least one sample key point information corresponding to each image; For any sample key point information, obtaining extraction ratio information according to the number of pictures from which the sample key point information is extracted in the first sample image set and the total number of pictures corresponding to the first sample image set; The key point extraction model is trained according to the key point information of each sample and the extraction ratio information corresponding to the key point information of each sample.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the information extraction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the information extraction method according to any one of claims 1 to 7 are implemented.

Images