Feature point extraction method and device, electronic equipment and readable medium

Abstract

The invention relates to a feature point extraction method and device, electronic equipment and a readable medium. The method comprises the following steps: acquiring a to-be-detected picture, and inputting the to-be-detected picture into a pre-trained feature network model; wherein the feature network model is obtained by training a target depth map output by a pre-trained depth network model; extracting feature information of each pixel point in the to-be-detected picture through the feature network model to obtain a target feature map; and performing conversion processing on the target feature map to obtain feature description information. According to the scheme provided by the invention, the problems of scale invariance limitation, view angle change sensitivity, shielding and deformation sensitivity, large-scale scene limitation, computing resource requirements, expression capability limitation of feature descriptors and the like can be deeply optimized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a feature point extraction method, device, electronic device and readable medium. Background Art

[0002] Image feature point extraction algorithms have wide applications in the field of computer vision. They are used to identify and describe local features in images, such as corners, edges, spots, etc. These features are unique and stable in images and can be used for tasks such as image matching, target recognition, image stitching, and 3D reconstruction.

[0003] In the related technologies, common image feature point extraction algorithms may include Harris (corner detection algorithm), SIFT (Scale-invariant feature transform), SURF (Speeded UpRobust Features), ORB (Oriented FAST and Rotated BRIEF), AKAZE (Accelerated-KAZE), BRISK (Binary Robust Invariant Scalable Keypoints), etc. These feature point extraction algorithms have been widely used in the field of computer vision, but there are also some common problems and limitations, such as scale invariance limitations, sensitivity to perspective changes, occlusion and deformation sensitivity, large-scale scene limitations, computing resource requirements, and limited expressiveness of feature descriptors. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a feature point extraction method, device, electronic device and readable medium, which can deeply optimize the scale invariance limitation, perspective change sensitivity, occlusion and deformation sensitivity, large-scale scene limitation, computing resource requirements, and feature descriptor expression ability limitation.

[0005] The first aspect of the present application provides a method for extracting feature points, the method comprising:

[0006] Obtain a picture to be detected, and input the picture to be detected into a pre-trained feature network model; wherein the feature network model is trained using a target depth map output by a pre-trained deep network model;

[0007] Extracting feature information of each pixel in the image to be detected through the feature network model to obtain a target feature map;

[0008] The target feature map is converted to obtain feature description information.

[0009] A second aspect of the present application provides a feature point extraction device, the device comprising:

[0010] The module for acquiring the image to be detected is used to acquire the image to be detected and input the image to be detected into a pre-trained feature network model; wherein the feature network model is trained by using the target depth map output by the pre-trained deep network model;

[0011] A first feature information extraction module is used to extract feature information of each pixel in the image to be detected through the feature network model to obtain a target feature map;

[0012] The conversion processing module is used to convert the target feature map to obtain feature description information.

[0013] A third aspect of the present application provides an electronic device, including:

[0014] Processor; and

[0015] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0017] The technical solution provided by this application may have the following beneficial effects:

[0018] The solution provided by the present application obtains the image to be detected and inputs the image to be detected into a pre-trained feature network model; wherein the feature network model is trained by using the target depth map output by the pre-trained deep network model; the feature network model extracts the feature information of each pixel in the image to be detected to obtain the target feature map; the target feature map is converted to obtain feature description information. The present application extracts feature points based on the method of convolutional neural network, which can deeply optimize the scale invariance limitation, sensitivity to perspective change, sensitivity to occlusion and deformation, large-scale scene limitation, computing resource requirements, and the expression ability limitation of feature descriptors.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0021] Figure 1 is a flow chart of a feature point extraction method shown in an embodiment of the present application;

[0022] Figure 2 is another flow chart of the feature point extraction method shown in an embodiment of the present application;

[0023] Figure 3 It is an application process architecture diagram of the characteristic network model shown in the embodiment of the present application;

[0024] Figure 4 It is a flowchart of a deep network model training method shown in an embodiment of the present application;

[0025] Figure 5 It is a training process architecture diagram of a deep network model and a feature network model shown in an embodiment of the present application;

[0026] Figure 6 It is an operation flow architecture diagram of standard inverse transformation and bilinear interpolation shown in an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of the operation of standard inverse transformation and bilinear interpolation shown in an embodiment of the present application;

[0028] Figure 8 It is a flowchart of a feature network model training method shown in an embodiment of the present application;

[0029] Fig. 9 is a structural schematic diagram of a feature point extraction device shown in an embodiment of the present application;

[0030] Fig.10 It is a structural diagram of a deep network model training device shown in an embodiment of the present application;

[0031] Fig.11 It is a structural schematic diagram of a feature network model training device shown in an embodiment of the present application;

[0032] Fig.12 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0034] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0036] Image feature point extraction algorithms have wide applications in the field of computer vision. They are used to identify and describe local features in images, such as corners, edges, spots, etc. These features are unique and stable in images and can be used for tasks such as image matching, target recognition, image stitching, and 3D reconstruction.

[0037] In the related technologies, common image feature point extraction algorithms and their application scenarios are as follows:

[0038] Harris (corner detection algorithm): Harris algorithm is a classic corner detection algorithm, which detects corners by calculating the corner response function of each pixel in the image. Harris algorithm is widely used in scenes such as image stitching, target tracking, and object recognition.

[0039] SIFT (Scale Invariant Feature Transform): SIFT is a feature point extraction and description algorithm based on scale space. It detects extreme points in the image at different scales and directions and constructs a descriptor to represent the local features of each feature point. SIFT is widely used in image matching, target recognition, 3D reconstruction and other scenarios.

[0040] SURF (Speeded Up Robust Feature) algorithm: SURF algorithm is an accelerated feature point extraction and description algorithm. It uses fast feature detection and descriptor calculation methods, so it has advantages in speed and robustness. SURF algorithm is widely used in real-time image processing, object recognition, image stitching and other scenarios.

[0041] ORB (Fast Feature Point Extraction and Description): The ORB algorithm is a method that combines FAST (Features from Accelerated Segment Test) feature detection and BRIEF (Binary Robust Independent Elementary Features) descriptors, which has a good balance between speed and performance. The ORB algorithm is widely used in real-time image processing, robot navigation, visual SLAM (Simultaneous Localization And Mapping) and other scenarios.

[0042] AKAZE (Fast Feature Point Extraction and Description Algorithm Based on Nonlinear Scale Space): AKAZE algorithm is a fast feature point extraction and description algorithm. It is based on KAZE algorithm (a feature point extraction algorithm based on nonlinear scale space) for accelerated optimization, so it performs well in speed and robustness. AKAZE algorithm is widely used in real-time image processing, image retrieval, 3D reconstruction and other scenarios.

[0043] BRISK (Binary Robust Invariant Feature Point Extraction Algorithm): BRISK algorithm is a binary robust invariant feature point extraction algorithm, which aims to provide an efficient and robust method for image feature point extraction. The design goal of BRISK algorithm is to combine speed, robustness and scalability. It adopts binary descriptors and uses scale space and direction invariance to extract and describe feature points in images. BRISK algorithm has been widely used in many computer vision applications, including object recognition, image stitching, real-time visual positioning, etc. It is suitable for scenarios with high speed requirements and certain robustness requirements, especially in embedded devices and real-time systems with limited computing resources.

[0044] The above algorithms are only a small part of the image feature point extraction algorithms. There are many other algorithms and variants. Choosing the appropriate algorithm depends on the needs of the specific application, including performance requirements, the type and number of image features, computing resources and other factors.

[0045] These image feature point extraction algorithms have been widely used in the field of computer vision, but they also have some common problems and limitations, as follows:

[0046] Scale invariance limitation: Feature point extraction methods in related technologies may perform poorly in situations where scale changes are large. For example, when objects appear at different scales in an image, feature point detection and matching may be affected.

[0047] Sensitivity to viewpoint changes: Feature point extraction methods in related technologies may not be stable for images with large viewpoint changes. For example, when an object is viewed from different angles, the position and descriptor of the feature points may change significantly, making matching difficult.

[0048] Occlusion and deformation sensitivity: The feature point extraction method in the related art is easily disturbed by images with occlusion or object deformation. For example, when the object is partially occluded or deformed, the detection and matching of feature points may fail or produce mismatches.

[0049] Large-scale scene limitations: The feature point extraction methods in related technologies are challenging for feature point detection and matching in large-scale scenes. For example, when the number of feature points in a scene is very large, the efficiency and accuracy of the algorithm may be limited.

[0050] Computational resource requirements: The feature point extraction methods in related technologies have high requirements in terms of computing resources, especially for real-time applications or devices with limited resources, which may face the problem of slow computing speed.

[0051] Limited expressive power of feature descriptors: The feature descriptors used in feature point extraction methods in related technologies are usually simple statistical or binary representations based on local pixel relationships, which may not accurately capture the semantic information of the image.

[0052] It can be seen that a method for improving feature point extraction is urgently needed in the related technology.

[0053] In response to the above problems, an embodiment of the present application provides a feature point extraction method, which extracts feature points by a method based on a convolutional neural network, and can deeply optimize problems such as scale invariance limitations, sensitivity to perspective changes, occlusion and deformation sensitivity, large-scale scene limitations, computing resource requirements, and the expressive power limitations of feature descriptors.

[0054] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0055] Figure 1 It is a flowchart of the feature point extraction method shown in the embodiment of the present application.

[0056] See also Figure 1 , the feature point extraction method of the present application comprises:

[0057] S110, obtaining a picture to be detected, and inputting the picture to be detected into a pre-trained feature network model; wherein the feature network model is trained by using a target depth map output by a pre-trained deep network model.

[0058] In the embodiment of the present application, the image to be detected may refer to an image that needs to be extracted with feature points, and its size is h*w*3, where h is the image height, w is the image width, and 3 is the RGB channel. The image to be detected can be acquired by a preset camera, where the preset camera can include at least one of a monocular camera, a binocular camera, and a multi-eye camera.

[0059] The embodiment of the present application can call a pre-trained feature network model (FeatureNet), and then input the image to be detected into the feature network model, so as to extract features from the image to be detected through the feature network model. Among them, the feature network model can be trained using a target depth map (DepthMap), and the target depth map can be obtained by outputting a pre-trained depth network model (DepthNet).

[0060] S120, extracting feature information of each pixel in the image to be detected through a feature network model to obtain a target feature map.

[0061] After the image to be detected is input into the feature network model, the feature network model can extract features of the input image to be detected. Specifically, the feature network model can extract feature information of each pixel in the image to be detected, and then output the extracted feature information of each pixel as a target feature map (FeatureMap).

[0062] S130, converting the target feature map to obtain feature description information.

[0063] The target feature map is stored in the form of an image, and its size is h*w*n, where n is the number of feature description dimensions, and n can be freely set according to the application. For example, n can be set to any value such as 16, 32, 64, etc.

[0064] The embodiment of the present application can convert the target feature map into feature description information. The feature description information is stored in the form of a vector with a size of Z*n, where Z is the number of rows contained in the feature description information, Z≤h*w, and each feature occupies each row in the feature description information, and each row has a feature description of n dimensions.

[0065] After obtaining the feature description information corresponding to the image to be detected, the feature description information can be used to perform tasks such as image matching, target recognition, image stitching, and three-dimensional reconstruction on the image to be detected. The embodiments of the present application do not limit this.

[0066] As can be seen from this example, the solution provided by this application obtains the image to be detected and inputs the image to be detected into a pre-trained feature network model; wherein the feature network model is trained using the target depth map output by the pre-trained deep network model; the feature network model extracts the feature information of each pixel in the image to be detected to obtain the target feature map; the target feature map is converted to obtain feature description information. This application extracts feature points based on the convolutional neural network method, which can deeply optimize scale invariance limitations, sensitivity to perspective changes, occlusion and deformation sensitivity, large-scale scene limitations, computing resource requirements, and limitations on the expressive power of feature descriptors.

[0067] Figure 2 It is another flowchart of the feature point extraction method shown in an embodiment of the present application.

[0068] See also Figure 2 , the feature point extraction method of the present application comprises:

[0069] S210, obtaining a picture to be detected, and inputting the picture to be detected into a pre-trained feature network model; wherein the feature network model is trained using a target depth map output by a pre-trained deep network model.

[0070] In the embodiment of the present application, the image to be detected may refer to an image that needs to be subjected to feature point extraction, and its size is h*w*3. The image to be detected may be acquired by a preset camera, wherein the preset camera may include at least one of a monocular camera, a binocular camera, and a multi-eye camera.

[0071] The embodiment of the present application can call a pre-trained feature network model, such as Figure 3 As shown, ImageC is a picture to be detected, FeatureNet is a pre-trained feature network model, and the embodiment of the present application can input the picture to be detected (ImageC) into the feature network model (FeatureNet) so as to extract features of the picture to be detected (ImageC) through the feature network model (FeatureNet). Among them, the feature network model (FeatureNet) can be trained using a target depth map (DepthMap), and the target depth map can be obtained by outputting a pre-trained depth network model (DepthNet).

[0072] S220, extracting feature information of each pixel in the image to be detected through a feature network model to obtain a target feature map.

[0073] like Figure 3As shown, after the image to be detected (ImageC) is input into the feature network model (FeatureNet), the feature network model (FeatureNet) outputs the corresponding FeatureMapC, and FeatureMapC is the target feature map. Specifically, the feature network model (FeatureNet) can extract the feature information of each pixel point in the image to be detected (ImageC), and then output the feature information of each pixel point extracted as the target feature map (FeatureMapC).

[0074] After obtaining the target feature map, if it is necessary to consider the scenario of computational overhead (post-processing), the embodiments of the present application can enter step S230; if it is not necessary to consider the scenario of computational overhead (post-processing), the embodiments of the present application can enter step S240.

[0075] S230, perform corner detection on the target feature map, and convert the feature information of the detected corners in the target feature map into feature description information.

[0076] The target feature map is stored in the form of an image, and its size is h*w*n. If all the feature information of all pixel points in the target feature map is output as vector-form feature description information, then the number of feature points to be matched subsequently is large, the computational amount is large, and the matching is time-consuming. Therefore, the embodiments of the present application can first perform corner detection on the target feature map. Among them, corners usually have clear and recognizable physical meanings. Using such feature points can ensure that the feature points extracted from different images have an intersection (physically the same point), which is beneficial to subsequent matching tasks.

[0077] The embodiments of the present application can use the Fast corner detection algorithm to perform corner detection on the target feature map, thereby reducing the computational amount of subsequent feature matching. As Figure 3 shown, the image to be detected (ImageC) passes through the feature network model (FeatureNet) to obtain the target feature map (FeatureMapC). At the same time, the embodiments of the present application can perform Fast corner detection on the image to be detected (ImageC) to obtain the corner position information, and then extract the feature information of the detected corners from the target feature map (FeatureMapC) based on the corner position information, and arrange it in vector form, thereby converting to obtain the feature description information (FeaturesC). Among them, the size of the feature description information converted after corner detection is Z*n, Z < h*w, so the computational amount of subsequent feature matching can be reduced.

[0078] Among them, the Fast corner point detection process can be: if a pixel point in the image to be detected has a specific attribute difference with enough continuous pixel points in its surrounding area, such as a grayscale difference, and the difference is greater than a specified threshold, it can be determined that the pixel point has a recognizable difference from its neighboring pixel points, so the pixel point can be used as a feature point (corner point).

[0079] S240, converting feature information of each pixel in the target feature map into feature description information.

[0080] like Figure 3 As shown in the figure, when there is no need to consider the scenario of computational overhead (post-processing), the Fast corner detection step can be omitted, and the entire target feature map (FeatureMapC) can be directly retained, so as to extract the feature information of all pixels in the target feature map (FeatureMapC), arrange them into a vector form, and then convert them into feature description information (FeaturesC). Among them, the size of the feature description information converted without corner detection is Z*n, Z=h*w.

[0081] After obtaining the feature description information corresponding to the image to be detected, the feature description information can be used to perform tasks such as image matching, target recognition, image stitching, and three-dimensional reconstruction on the image to be detected. The embodiments of the present application do not limit this.

[0082] As can be seen from this example, the solution provided by the present application obtains the image to be detected and inputs the image to be detected into a pre-trained feature network model; wherein the feature network model is obtained by training the target depth map output by the pre-trained deep network model; the feature information of each pixel in the image to be detected is extracted by the feature network model to obtain the target feature map; the target feature map is subjected to corner point detection, and the feature information of the detected corner points in the target feature map is converted into feature description information; or, the feature information of each pixel in the target feature map is converted into feature description information. The present application extracts feature points based on the method of convolutional neural network, which can deeply optimize the scale invariance limitation, the sensitivity of viewing angle change, the sensitivity of occlusion and deformation, the limitation of large-scale scene, the requirement of computing resources, the limitation of the expression ability of feature descriptors, and other issues. Furthermore, the present application can reduce the computational amount of feature matching by performing corner point detection on the target feature map and then converting the feature information of the detected corner points into feature description information; further, when there is no need to consider the scene of computational overhead (post-processing), the feature information of all pixels in the entire target feature map can be directly retained and all converted into feature description information.

[0083] Figure 4 It is a flowchart of the deep network model training method shown in an embodiment of the present application.

[0084] See also Figure 4 , the deep network model training method of the present application comprises:

[0085] S410, obtaining a plurality of sample images; wherein the plurality of sample images are images captured by a preset camera at different viewing angles.

[0086] The embodiment of the present application may include two training stages, the first training stage is used to train the deep network model, and the second training stage is used to train the feature network model. Figure 5 As shown in the figure, the dark background part is the network that needs to be trained: DepthNet and FeatureNet, where DepthNet is a deep network model and FeatureNet is a feature network model. The dark thick link is the first training stage. The first training stage uses Loss1 and only trains DepthNet. When the first stage is trained to the point where Loss1 no longer decreases (converges), it enters the second training stage. In the second training stage, only FeatureNet is trained, that is, the last two steps of the dark thick link (dashed part) are no longer executed, and the back propagation of Loss2 will not update the network parameters of DepthNet, but only update the network parameters of FeatureNet.

[0087] Steps S410 to S470 are the first training stage. Specifically, the embodiment of the present application can first collect multiple pictures of different viewing angles through a preset camera, and then use these pictures as sample pictures (Image), wherein the size of each sample picture can be h*w*3.

[0088] The preset camera may include at least one of a monocular camera, a binocular camera, and a multi-eye camera. A monocular camera may be composed of one visual sensor (camera), a binocular camera may be composed of two visual sensors, and a multi-eye camera may be composed of three or more visual sensors. Therefore, if the image is captured by a monocular camera, the sample image may refer to the image captured by the monocular camera at different times, and the images captured at different times have different perspectives; if the image is captured by a binocular camera or a multi-eye camera, the sample image may refer to the left and right frames of the image captured by the binocular camera or the multi-eye camera at the same time, and the left and right frames of the image captured at the same time have different perspectives; if the image is captured by a binocular camera or a multi-eye camera, the sample image may refer to the image captured by the binocular camera or the multi-eye camera at different times, and the images captured at different times have different perspectives.

[0089] Among them, different viewing angles may be different angles, different distances, or different heights, which is not limited in the embodiments of the present application.

[0090] S420: Select two sample pictures separated by a preset number of frames from the plurality of sample pictures.

[0091] During each training, two sample images separated by a preset number of frames can be selected from multiple sample images. The two selected images can be two adjacent images, that is, the preset number of frames is 0; the two selected images can also be two non-adjacent images, that is, the preset number of frames is ≥ 1. Preferably, the selection condition is: the acquisition distance between the two sample images is less than or equal to the preset distance, and the two sample images contain more than 2 / 3 of the same area.

[0092] S430: Input the two sample images into the deep network model, so as to respectively extract the depth information of each pixel in the two sample images through the deep network model to obtain two sample depth maps.

[0093] In an embodiment of the present application, two selected sample images can be input into a deep network model so that the depth information of each pixel in each sample image can be extracted through the deep network model (DepthNet), and then the depth information of each pixel extracted from each sample image can be output as a sample depth map (DepthMap), wherein the size of each sample depth map is the same as the sample image, that is, the size of each sample depth map is also h*w.

[0094] In one example, if Figure 5 As shown, ImageS and ImageT are respectively selected sample images. In the embodiment of the present application, ImageS can be input into DepthNet, the depth information of each pixel in ImageS can be extracted through DepthNet, and then the depth information of each extracted pixel can be output as DepthMapS. Similarly, ImageT can be input into DepthNet, the depth information of each pixel in ImageT can be extracted through DepthNet, and then the depth information of each extracted pixel can be output as DepthMapT. Among them, DepthMapS is the sample depth map corresponding to ImageS, and DepthMapT is the sample depth map corresponding to ImageT.

[0095] S440, obtaining camera intrinsic parameters and camera extrinsic parameters of a preset camera.

[0096] In one example, if the two sample images selected are left and right frames captured by a binocular camera or a multi-camera at the same time, the camera intrinsic parameters and camera extrinsic parameters of the binocular camera or the multi-camera are usually pre-calibrated, so the calibrated camera intrinsic parameters and camera extrinsic parameters can be directly obtained. Figure 5 As shown in the figure, assume that ImageS and ImageT are the left and right frames captured by the binocular camera at the same time.s,t and t s,t is the extrinsic parameter matrix of the calibrated binocular camera, K s and K t are the intrinsic parameter matrices of the binocular cameras, so the calibrated camera intrinsic parameters (K s and K t ) and camera external parameters (R s,t and t s,t ).

[0097] In another example, if the two selected sample images are two frames of images captured by a preset camera (monocular camera, binocular camera, or multi-camera) at different times, the deep network model (DepthNet) can output the camera intrinsic parameters and camera extrinsic parameters corresponding to each sample image while outputting the sample depth map (DepthMap) corresponding to each sample image. Therefore, the camera intrinsic parameters and camera extrinsic parameters output by the deep network model (DepthNet) can be directly obtained. Figure 5 As shown in the figure, assuming that ImageS and ImageT are two frames of images collected by a monocular camera at different times, DepthNet can output R synchronously while outputting DepthMapS and DepthMapT. s,t and t s,t , and K s =K t , so the camera internal parameters (K) output by DepthNet can be directly obtained s and K t ) and camera external parameters (R s,t and t s,t ).

[0098] S450, based on the camera intrinsic parameters and the camera extrinsic parameters, respectively perform standard inverse transformation operations on the two sample depth maps to obtain two sample mapping maps.

[0099] After obtaining the camera intrinsic parameters and camera extrinsic parameters, a standard inverse transformation operation can be performed on each sample depth map (DepthMap) based on the camera intrinsic parameters and camera extrinsic parameters to obtain a sample mapping map (WarpMap), wherein the sample mapping map (WarpMap) can be used to characterize the corresponding pixel positions of each pixel point in a sample image in other images.

[0100] In one embodiment, the two sample images include a first sample image captured based on a first viewing angle and a second sample image captured based on a second viewing angle, and the sample depth map includes a first sample depth map corresponding to the first sample image and a second sample depth map corresponding to the second sample image; based on camera intrinsic parameters and camera extrinsic parameters, the two sample depth maps are respectively subjected to standard inverse transformation operations to obtain two sample mapping maps, which may include:

[0101] Based on the camera intrinsic parameters and the camera extrinsic parameters, a standard inverse transformation operation is performed on the first sample depth map to obtain a first sample mapping map; and based on the camera intrinsic parameters and the camera extrinsic parameters, a standard inverse transformation operation is performed on the second sample depth map to obtain a second sample mapping map; wherein the first sample mapping map is used to represent that each first pixel point in the first sample image is respectively mapped to the first pixel position in the second sample image, and the second sample mapping map is used to represent that each second pixel point in the second sample image is respectively mapped to the second pixel position in the first sample image.

[0102] The two selected sample images are images captured based on different viewing angles, where the different viewing angles may include a first viewing angle and a second viewing angle. For ease of description, the sample image captured based on the first viewing angle may be referred to as the first sample image, and the sample image captured based on the second viewing angle may be referred to as the second sample image. For example, Figure 5 As shown, assuming that ImageS is a sample image captured based on the first perspective, and ImageT is a sample image captured based on the second perspective, ImageS can be used as the first sample image, and ImageT can be used as the second sample image.

[0103] The first sample image is output through the deep network model to obtain a corresponding first sample depth map, and the second sample image is output through the deep network model to obtain a corresponding second sample depth map. Figure 5 As shown, DepthNet can output a first sample depth map (DepthMapS) corresponding to a first sample image (ImageS), and can output a second sample depth map (DepthMapT) corresponding to a second sample image (ImageT).

[0104] In the first training phase, Figure 5 As shown, the first sample depth map (DepthMapS) and the second sample depth map (DepthMapT) enter the dark thick link (dashed line part) process, in which It is called the standard inverse transform operation. The standard inverse transform operation process can be found in Figure 6 .like Figure 6 As shown in the figure, the internal and external parameters of the camera are the camera internal parameters (K s and K t ) and camera external parameters (R s ,t and t s ,t), the embodiment of the present application can be based on the camera internal parameter (K s and K t ) and camera external parameters (R s,t and t s,t), a standard inverse transform operation is performed on the first sample depth map (DepthMapS), and a corresponding first sample mapping map (WarpMapT) can be obtained, wherein the first sample mapping map (WarpMapT) can be used to characterize that each first pixel point in the first sample image (ImageS) is respectively mapped to the first pixel position in the second sample image (ImageT), the first pixel point is the pixel point in the first sample image, and the first pixel position of a first pixel point is the pixel position corresponding to the first pixel point in the first sample image in the second sample image. Similarly, the embodiment of the present application can be based on the camera intrinsic parameter (K s and K t ) and camera external parameters (R s,t and t s,t ), a standard inverse transform operation is performed on the second sample depth map (DepthMapT) to obtain a corresponding second sample mapping map (WarpMapS), wherein the second sample mapping map (WarpMapS) can be used to characterize that each second pixel point in the second sample image (ImageT) is respectively mapped to a second pixel position in the first sample image (ImageS), the second pixel point is the pixel point in the second sample image, and the second pixel position of a second pixel point is the pixel position corresponding to the second pixel point in the second sample image in the first sample image.

[0105] Taking the standard inverse transform operation process of the second sample depth map (DepthMapT) as an example, the calculation process of the standard inverse transform operation is as follows:

[0106] From the camera imaging formula (Formula 1), (Formula 2) can be derived, which is called the standard inverse homography.

[0107] Formula 1:

[0108]

[0109] Among them, P world represents the spatial coordinates (x, y, z) of a point in the world coordinate system, [u, v] represents the pixel position of the point in the camera image, and R cam,world and t cam,world Represents the coordinate transformation matrix between the world coordinate system and the camera coordinate system, corresponding to the rotation (3x3) matrix and the translation (3x1) matrix, K represents the camera intrinsic parameter matrix, and d represents P world The z coordinate value in the camera coordinate system is also called depth.

[0110] Formula 2:

[0111]

[0112] Where m = [0, 0, 1], the parameters with the subscript s represent the parameters under the source camera, and the parameters with the subscript t represent the parameters under the target camera. t ,v t ] represents the pixel position of the second pixel point q in the second sample image (ImageT), d t Represents the value of the corresponding position of the second pixel point q in the second sample depth map (DepthMapT), Represents R s,t The transposed matrix of s,t Represents the rotation matrix from the target camera coordinate system to the source camera coordinate system, K s and K t represents the camera intrinsic parameter matrix, [u s ',v s '] represents the pixel position ([ut, vt]) of the second pixel point q in the second sample image (ImageT) corresponding to the pixel position in the first sample image (ImageS), that is, [ut s ',v s '] is the second pixel position, the second pixel position [u s ',v s '] is usually a decimal (not an integer).

[0113] It should be noted that the source camera refers to the camera that collects the first sample image (ImageS), and the target camera refers to the camera that collects the second sample image (ImageT). If the image is collected by a monocular camera, the source camera and the target camera are both monocular cameras; if the image is collected by a binocular camera, the source camera and the target camera are the left camera and the right camera respectively; if the image is collected by a multi-camera camera, the source camera and the target camera are any two cameras.

[0114] By using the above formula 2 and the second sample depth map (DepthMapT), the pixel position ([u t ,v t ]) is mapped to the second pixel position q'([u s ',v s ']). Among them, a Map of size h*w*2 can be generated, and then at the pixel position of the second pixel point q in the Map ([u t ,v t ]) fills in the second pixel position q'([u s ',v s']), the second sample map (WarpMapS) can be obtained.

[0115] Similarly, the first sample map (WarpMapT) can be obtained through the above formula 2 and the first sample depth map (DepthMapS).

[0116] S460, using two sample mapping images to train the deep network model to obtain a deep loss value.

[0117] After obtaining the first sample mapping map (WarpMapT) and the second sample mapping map (WarpMapS), the first sample mapping map (WarpMapT) and the second sample mapping map (WarpMapS) can be used to train the deep network model (DepthNet) to obtain the depth loss value (Loss1) of this training.

[0118] In one embodiment, using two sample maps to train a deep network model to obtain a deep loss value may include:

[0119] According to the two sample mapping images, bilinear interpolation is performed on the two sample images to obtain two predicted images; based on the two sample images and the two predicted images, the depth loss value of this training is calculated.

[0120] The embodiment of the present application can interpolate two sample images (Image) according to two sample mapping images (WarpMap), for example, bilinear interpolation, to obtain two corresponding prediction images (WarpImage), wherein the size of each prediction image is the same as the sample image, that is, the size of the prediction image is also h*w.

[0121] like Figure 6 As shown, the first sample mapping map (WarpMapT) represents that each first pixel point in the first sample image (ImageS) is mapped to the first pixel position in the second sample image (ImageT), so the first sample mapping map (WarpMapT) and the second sample image (ImageT) can be used to calculate a new image through a bilinear interpolation method, and the new image can be used as the first predicted image (WarpImageT). The first predicted image (WarpImageT) is the first sample image (ImageS) predicted by bilinear interpolation based on the second sample image (ImageT), so in an ideal state, the first predicted image (WarpImageT) is exactly the same as the first sample image (ImageS).

[0122] like Figure 6As shown, the second sample mapping map (WarpMapS) represents that each second pixel point in the second sample image (ImageT) is mapped to the second pixel position in the first sample image (ImageS), so the second sample mapping map (WarpMapS) and the first sample image (ImageS) can be used to calculate a new image through a bilinear interpolation method, and the new image can be used as the second predicted image (WarpImageS). The second predicted image (WarpImageS) is the second sample image (ImageT) predicted by bilinear interpolation based on the first sample image (ImageS), so in an ideal state, the second predicted image (WarpImageS) is exactly the same as the second sample image (ImageT).

[0123] The ideal state may refer to that each variable involved in the calculation is a true value, and factors such as illumination and brightness do not change with the camera's acquisition viewing angle.

[0124] However, it is difficult to achieve the ideal state in practical applications. For example, two pictures taken from different perspectives will always have different lighting or brightness. Therefore, the depth loss value (Loss1) of this training can be calculated based on two sample pictures (Image) and two predicted pictures (WarpImage).

[0125] In one implementation, bilinear interpolation is performed on two sample images according to two sample mapping images to obtain two predicted images, which may include:

[0126] Extract each first pixel position from the first sample mapping image, and extract each second pixel position from the second sample mapping image; search for multiple second pixel points adjacent to each first pixel position in the second sample image, and search for multiple first pixel points adjacent to each second pixel position in the first sample image; perform bilinear interpolation on the pixel intensities of the multiple second pixel points to obtain the pixel intensities of each first pixel position, and perform bilinear interpolation on the pixel intensities of the multiple first pixel points to obtain the pixel intensities of each second pixel position; assign the pixel intensities of each first pixel position to the corresponding first pixel point in the first sample mapping image to obtain a first predicted image, and assign the pixel intensities of each second pixel position to the corresponding second pixel point in the second sample mapping image to obtain a second predicted image.

[0127] Among them, bilinear interpolation is also called bilinear interpolation. Mathematically, bilinear interpolation is an extension of linear interpolation with an interpolation function of two variables. Its core idea is to perform linear interpolation in two directions respectively.

[0128] Take the bilinear interpolation process of the first sample image (ImageS) as an example: Figure 7As shown, the second sample mapping map (WarpMapS) can represent that the pixel position of the second pixel point q in the second sample image (ImageT) is mapped to the second pixel position q' in the first sample image (ImageS). Therefore, the second pixel position q' can be extracted from the second sample mapping map (WarpMapS), and then multiple first pixel points adjacent to the second pixel position q' can be found in the first sample image (ImageS). For example, four first pixel points adjacent to the second pixel position q' can be found: q lt ',q rt ',q lb ',q rb ', and then the pixel intensity of the second pixel position q' can be obtained by bilinear interpolation. Specifically, linear interpolation can be performed in the x direction to obtain the first pixel points R1 and R2, and then linear interpolation can be performed in the y direction to obtain the first pixel point P. For example, Figure 7 As shown, the x direction includes two pairs of first pixel points, through one pair of first pixel points (q lt 'and q rt ') insert a first pixel point R1 between the two pairs, and insert a first pixel point R2 between the two pairs of first pixel points (q lb 'and q rb '), and then a first pixel point P is calculated by interpolating the first pixel point R1 and the first pixel point R2 in the y direction, thereby obtaining the pixel intensity of the second pixel position q': f(x, y). In the embodiment of the present application, the pixel intensity of the second pixel position q' can be assigned to the corresponding second pixel point q in the second sample map (WarpMapS), thereby assigning a new pixel intensity to the second pixel point q.

[0129] According to the above bilinear interpolation method, the entire second sample map (WarpMapS) is traversed to calculate the pixel intensities of all second pixels, and the second predicted image (WarpImageS) can be obtained.

[0130] Similarly, by traversing the entire first sample map (WarpMapT) in the above-mentioned bilinear interpolation manner, the pixel intensities of all first pixels can be calculated to obtain the first predicted image (WarpImageT).

[0131] In one implementation, calculating the depth loss value of this training according to the two sample images and the two prediction images may include:

[0132] The difference between the first sample image and the first predicted image is calculated to obtain the depth loss value of the first perspective, the difference between the second sample image and the second predicted image is calculated to obtain the depth loss value of the second perspective, and the error value of the camera extrinsic parameter is calculated; the depth loss value of the first perspective, the depth loss value of the second perspective and the error value of the camera extrinsic parameter are added to obtain the depth loss value of this training.

[0133] In an ideal state, the first predicted image (WarpImageT) is exactly the same as the first sample image (ImageS), and the second predicted image (WarpImageS) is exactly the same as the second sample image (ImageT). However, it is difficult to achieve the ideal state in practical applications. For example, two images captured from different perspectives always have different illumination or brightness. Therefore, the embodiment of the present application can calculate the difference between the first sample image (ImageS) and the first predicted image (WarpImageT), and the difference is the depth loss value of the first perspective. The difference can also be calculated between the second sample image (ImageT) and the second predicted image (WarpImageS), and the difference is the depth loss value of the second perspective. Then, the depth loss value of the first perspective can be added to the depth loss value of the second perspective to obtain the depth loss values ​​of different perspectives (Loss vs ), where Loss vs The specific calculation is shown in Formula 3.

[0134] Formula 3:

[0135] Loss vs =norm2(ImageS-WarpImageT)+norm2(ImageT-WarpImageS)

[0136] In addition, the embodiment of the present application can calculate the camera external parameters (R s,t and t s,t ) error value (Loss pose Specifically, case 1: if the first sample image (ImageS) and the second sample image (ImageT) are left and right frames captured by a binocular camera or a multi-camera at the same time, then the camera extrinsic parameter (R s,t and t s,t ) is usually pre-calibrated. Since the current calibration accuracy is relatively high, the embodiment of the present application can fully trust the calibrated camera external parameters (R s,t and t s,t ), so there is no need to calculate the camera extrinsic parameters (R s,t and t s,t ) error value (Loss pose). Case 2: If the first sample image (ImageS) and the second sample image (ImageT) are images captured by a preset camera (monocular camera, binocular camera, or multi-camera) at different times, then during the image acquisition process, the pose true value (i.e. and ), then the embodiment of the present application can also fully trust the true value of the pose without calculating Loss pose , and you can also calculate Loss pose , in this case, calculate Loss pose It can accelerate the convergence of the deep network model (DepthNet). Case 3: In case 2, if the pose true value is not collected, the Loss cannot be calculated. pose .

[0137] Among them, Loss pose The specific calculation is shown in Formula 4.

[0138] Formula 4:

[0139]

[0140] Among them, Rodrigues is the Rodriguez formula, which can convert a 3x3 rotation matrix into a 3x1 rotation vector, and norm2 is the 2 norm, that is, the modulus length. and It is the pose truth value collected by centimeter-level RTK.

[0141] In addition, the embodiment of the present application can also calculate other error values ​​(Loss other ), Loss other It can be used to accelerate the convergence of deep network models (DepthNet).

[0142] In the embodiment of the present application, the depth loss values ​​(Loss vs ), the error value of the camera external parameters (Loss pose ) and other error values ​​(Loss other ) are added to obtain the depth loss value (Loss1) of this training, where the specific calculation of Loss1 is shown in Formula 5.

[0143] Formula 5:

[0144] Loss1=Loss vs +Loss pose +Loss other

[0145] S470, using the depth loss value to reversely update the network parameters of the deep network model until the depth loss value is less than or equal to the preset depth loss threshold, thereby obtaining a trained deep network model.

[0146] After obtaining the depth loss value (Loss1) of this training, it can be determined whether the depth loss value (Loss1) of this training is less than or equal to the preset depth loss threshold. If the depth loss value (Loss1) of this training is greater than the preset depth loss threshold, it means that the depth loss value (Loss1) of this training can continue to decrease, that is, the deep network model (DepthNet) is not currently converged, then the embodiment of the present application can use the depth loss value (Loss1) of this training, reversely update the network parameters of the deep network model (DepthNet), and then return to step S420 to continue training; if the depth loss value (Loss1) of this training is less than or equal to the preset depth loss threshold, it means that the depth loss value (Loss1) of this training no longer decreases, that is, the deep network model (DepthNet) is currently converged, then the embodiment of the present application can obtain a trained deep network model (DepthNet).

[0147] In an example, assuming there are 100 sample images (Image1 to Image100), if three adjacent images contain more than 2 / 3 of the same area, for example, Image1, Image2 and Image3 contain more than 2 / 3 of the same area, then Image1 and Image2 can be used as training group 1, Image1 and Image3 as training group 2, and Image2 and Image3 as training group 3. In the first training stage, the DepthNet is first trained with training group 1 (Image1 and Image2) to obtain the Loss1 of the first training. If the Loss1 of the first training is greater than the preset depth loss threshold, the Loss1 of the first training is used to reversely update the network parameters of the DepthNet; then, the DepthNet is trained with training group 2 (Image1 and Image3) to obtain the Loss1 of the second training. If the Loss1 of the second training is greater than the preset depth loss threshold, the Loss1 of the second training is used to reversely update the network parameters of the DepthNet; then, the DepthNet is trained with training group 3 (Image2 and Image3) to obtain the Loss1 of the third training. If the Loss1 of the third training is greater than the preset depth loss threshold, the Loss1 of the third training is used to reversely update the network parameters of the DepthNet; ...; until the Loss1 of the Nth training is less than the preset depth loss threshold, the trained DepthNet is obtained. Where N is a positive integer.

[0148] Figure 8 It is a flowchart of the feature network model training method shown in an embodiment of the present application.

[0149] See also Figure 8 , the feature network model training method of the present application includes:

[0150] S810, obtaining a plurality of sample images; wherein the plurality of sample images are images captured by a preset camera at different viewing angles.

[0151] like Figure 5 As shown in the figure, when the first stage training reaches the point where Loss1 no longer decreases (converges), the second training stage is entered. In the second training stage, only FeatureNet is trained, that is, the last two steps (dashed line part) of the dark thick link are no longer executed, and the back propagation of Loss2 does not update the network parameters of DepthNet, but only updates the network parameters of FeatureNet.

[0152] The steps S810 to S870 are the second training stage, wherein the multiple sample images obtained in step S810 are the same as the multiple sample images obtained in step S410. This step can refer to the description in S410 and will not be repeated here.

[0153] S820: Select two sample pictures separated by a preset number of frames from the multiple sample pictures.

[0154] For this step, please refer to the description in S820 and will not be repeated here.

[0155] S830: Input the two sample images into a pre-trained deep network model, so as to extract the depth information of each pixel in the two sample images through the deep network model, and obtain two target depth maps.

[0156] In the second training stage, the deep network model (DepthNet) trained in the first training stage can be called, and then the two sample images can be input into the trained deep network model (DepthNet), so that the depth information of each pixel in each sample image can be extracted through the trained deep network model (DepthNet), and then the depth information of each pixel extracted from each sample image is output as a target depth map (DepthMap), wherein the size of each target depth map is the same as that of the sample image, that is, the size of each target depth map is also h*w.

[0157] It should be noted that both the sample depth map and the target depth map are depth maps output by the deep network model (DepthNet). The difference is that the sample depth map is a depth map output by a deep network model (DepthNet) that has not yet been trained, and the target depth map is a depth map output by a deep network model (DepthNet) that has been trained.

[0158] S840, obtaining camera intrinsic parameters and camera extrinsic parameters of a preset camera.

[0159] For this step, please refer to the description in S440 and will not be repeated here.

[0160] S850, based on the camera intrinsic parameters and the camera extrinsic parameters, perform standard inverse transformation operations on the two target depth maps respectively to obtain two target mapping maps.

[0161] After obtaining the camera intrinsic parameters and camera extrinsic parameters, a standard inverse transformation operation can be performed on each target depth map (DepthMap) based on the camera intrinsic parameters and camera extrinsic parameters to obtain a target mapping map (WarpMap), wherein the target mapping map (WarpMap) can be used to characterize the corresponding pixel positions of each pixel point in a sample image in other images.

[0162] In one embodiment, the two sample images include a first sample image captured based on a first perspective and a second sample image captured based on a second perspective, and the target depth map includes a first target depth map corresponding to the first sample image and a second target depth map corresponding to the second sample image; based on camera intrinsic parameters and camera extrinsic parameters, the two target depth maps are respectively subjected to standard inverse transformation operations to obtain two target mapping maps, which may include:

[0163] Based on the camera intrinsic parameters and the camera extrinsic parameters, a standard inverse transformation operation is performed on the first target depth map to obtain a first target mapping map; and based on the camera intrinsic parameters and the camera extrinsic parameters, a standard inverse transformation operation is performed on the second target depth map to obtain a second target mapping map; wherein the first target mapping map is used to represent that each first pixel point in the first sample image is respectively mapped to the first pixel position in the second sample image, and the second target mapping map is used to represent that each second pixel point in the second sample image is respectively mapped to the second pixel position in the first sample image.

[0164] like Figure 5As shown, the two sample images include a first sample image (ImageS) captured based on a first perspective and a second sample image (ImageT) captured based on a second perspective. The pre-trained DepthNet can output a first target depth map (DepthMapS) corresponding to the first sample image (ImageS), and can output a second target depth map (DepthMapT) corresponding to the second sample image (ImageT).

[0165] In the second training phase, Figure 5 As shown, the first target depth map (DepthMapS) and the second target depth map (DepthMapT) enter the light-colored thin link process, in which It is called the standard inverse transform operation. The standard inverse transform operation process can be found in Figure 6 .like Figure 6 As shown, the embodiment of the present application can be based on the camera internal parameter (K s and K t ) and camera external parameters (R s,t and t s,t ), a standard inverse transform operation is performed on the first target depth map (DepthMapS), and a corresponding first target mapping map (WarpMapT) can be obtained, wherein the first target mapping map (WarpMapT) can be used to represent that each first pixel point in the first sample image (ImageS) is mapped to the first pixel position in the second sample image (ImageT). Similarly, the embodiment of the present application can be based on the camera intrinsic parameter (K s and K t ) and camera external parameters (R s,t and t s,t ), a standard inverse transform operation is performed on the second target depth map (DepthMapT) to obtain a corresponding second target mapping map (WarpMapS), wherein the second target mapping map (WarpMapS) can be used to characterize that each second pixel point in the second sample image (ImageT) is mapped to the second pixel position in the first sample image (ImageS).

[0166] The standard inverse transformation operation process of the first target depth map (DepthMapS) and the second target depth map (DepthMapT) is the same as the standard inverse transformation operation process of the second sample depth map (DepthMapT) described above, and will not be repeated here.

[0167] S860, two target mapping images are used to train the feature network model to obtain the feature loss value.

[0168] After obtaining the first target mapping map (WarpMapT) and the second target mapping map (WarpMapS), the first target mapping map (WarpMapT) and the second target mapping map (WarpMapS) can be used to train the feature network model (FeatureNet) to obtain the feature loss value (Loss2) of this training.

[0169] In one embodiment, using two target maps to train the feature network model to obtain a feature loss value may include:

[0170] The two sample images are input into the feature network model to extract the feature information of each pixel in the two sample images through the feature network model to obtain two sample feature maps; according to the two target mapping maps, the two sample feature maps are bilinearly interpolated to obtain two predicted feature maps; according to the two sample feature maps and the two predicted feature maps, the feature loss value of this training is calculated.

[0171] Two sample images can be input into a feature network model (FeatureNet) so that feature information of each pixel in each sample image can be extracted through the feature network model (FeatureNet), and then the feature information of each pixel extracted from each sample image is output as a sample feature map (FeatureMap), wherein the size of each sample feature map is the same as the target feature map, that is, the size of each sample feature map is also h*w*n.

[0172] The embodiment of the present application can interpolate two sample feature maps (FeatureMap) according to two target mapping maps (WarpMap), such as bilinear interpolation, to obtain two corresponding prediction feature maps (WarpFeatureMap), wherein the size of each prediction feature map is the same as the sample feature map, that is, the size of each prediction feature map is also h*w*n.

[0173] like Figure 6As shown, the first target map (WarpMapT) represents that each first pixel point in the first sample image (ImageS) is mapped to the first pixel position in the second sample image (ImageT), so the first target map (WarpMapT) and the second sample feature map (FeatureMapT) can be used to calculate a new feature map through a bilinear interpolation method, and the new feature map can be used as the first predicted feature map (WarpFeatureMapT). Among them, the first predicted feature map (WarpFeatureMapT) is the first sample feature map (FeatureMapS) predicted by bilinear interpolation based on the second sample feature map (FeatureMapT), so in an ideal state, the first predicted feature map (WarpFeatureMapT) is exactly the same as the first sample feature map (FeatureMapS).

[0174] like Figure 6 As shown, the second target map (WarpMapS) represents that each second pixel point in the second sample image (ImageT) is mapped to the second pixel position in the first sample image (ImageS), so the second target map (WarpMapS) and the first sample feature map (FeatureMapS) can be used to calculate a new feature map through a bilinear interpolation method, and the new feature map can be used as the second predicted feature map (WarpFeatureMapS). Among them, the second predicted feature map (WarpFeatureMapS) is the second sample feature map (FeatureMapT) predicted by bilinear interpolation based on the first sample feature map (FeatureMapS), so in an ideal state, the second predicted feature map (WarpFeatureMapS) is exactly the same as the second sample feature map (FeatureMapT).

[0175] The ideal state may refer to that each variable involved in the calculation is a true value, and factors such as illumination and brightness do not change with the camera's acquisition viewing angle.

[0176] However, it is difficult to achieve the ideal state in practical applications. For example, two pictures taken from different perspectives will always have different lighting or brightness. Therefore, the feature loss value (Loss2) of this training can be calculated based on two sample feature maps (FeatureMap) and two predicted feature maps (WarpFeatureMap).

[0177] In one embodiment, the two sample feature maps include a first sample feature map corresponding to the first sample image and a second sample feature map corresponding to the second sample image; and according to the two target mapping maps, bilinear interpolation is performed on the two sample feature maps respectively to obtain two predicted feature maps, which may include:

[0178] Extract each first pixel position from the first target mapping map, and extract each second pixel position from the second target mapping map; search for multiple second pixel points adjacent to each first pixel position in the second sample feature map, and search for multiple first pixel points adjacent to each second pixel position in the first sample feature map; perform bilinear interpolation on the pixel intensities of the multiple second pixel points to obtain the pixel intensities of each first pixel position, and perform bilinear interpolation on the pixel intensities of the multiple first pixel points to obtain the pixel intensities of each second pixel position; assign the pixel intensities of each first pixel position to the corresponding first pixel point in the first target mapping map to obtain a first predicted feature map, and assign the pixel intensities of each second pixel position to the corresponding second pixel point in the second target mapping map to obtain a second predicted feature map.

[0179] like Figure 5 As shown, the two sample feature maps include a first sample feature map (FeatureMapS) corresponding to the first sample image (ImageS) and a second sample feature map (FeatureMapT) corresponding to the second sample image (ImageT).

[0180] The bilinear interpolation process of the first sample feature map (FeatureMapS) and the second sample feature map (FeatureMapT) is the same as the bilinear interpolation process of the first sample image (ImageS) mentioned above, and will not be repeated here.

[0181] In one implementation, calculating the feature loss value of this training according to the two sample feature maps and the two predicted feature maps may include:

[0182] The difference between the first sample feature map and the first predicted feature map is calculated to obtain the feature loss value of the first perspective, and the difference between the second sample feature map and the second predicted feature map is calculated to obtain the feature loss value of the second perspective; the feature loss value of the first perspective is added to the feature loss value of the second perspective to obtain the feature loss value of this training.

[0183] Under ideal conditions, the first predicted feature map (WarpFeatureMapT) is exactly the same as the first sample feature map (FeatureMapS), and the second predicted feature map (WarpFeatureMapS) is exactly the same as the second sample feature map (FeatureMapT). However, it is difficult to achieve the ideal state in practical applications. For example, two pictures captured from different perspectives always have differences in illumination or brightness. Therefore, the embodiment of the present application can calculate the difference between the first sample feature map (FeatureMapS) and the first predicted feature map (WarpFeatureMapT), and the difference is the feature loss value of the first perspective, and can calculate the difference between the second sample feature map (FeatureMapT) and the second predicted feature map (WarpFeatureMapS), and the difference is the feature loss value of the second perspective. Then, the feature loss value of the first perspective and the feature loss value of the second perspective can be added to obtain the feature loss value (Loss2) of this training, where the specific calculation of Loss2 is shown in Formula 6.

[0184] Formula 6:

[0185] Loss2=norm2(FeatureMapS-WarpFeatureMapT)+norm2(FeatureMapT-WarpFeatureMapS)

[0186] S870, using the feature loss value, reversely updating the network parameters of the feature network model until the feature loss value is less than or equal to the preset feature loss threshold, thereby obtaining a trained feature network model.

[0187] After obtaining the feature loss value (Loss2) of this training, it can be determined whether the feature loss value (Loss2) of this training is less than or equal to the preset feature loss threshold. If the feature loss value (Loss2) of this training is greater than the preset feature loss threshold, it means that the feature loss value (Loss2) of this training can continue to decrease, that is, the feature network model (FeatureNet) is not currently converged, then the embodiment of the present application can use the feature loss value (Loss2) of this training, reversely update the network parameters of the feature network model (FeatureNet), and then return to step S820 to continue training; if the feature loss value (Loss2) of this training is less than or equal to the preset feature loss threshold, it means that the feature loss value (Loss2) of this training no longer decreases, that is, the feature network model (FeatureNet) is currently converged, then the embodiment of the present application can obtain a trained feature network model (FeatureNet).

[0188] In one example, similar to the first training stage, the second training stage also has 100 sample images (Image1 to Image100), and the three adjacent images contain more than 2 / 3 of the same area. For example, Image1, Image2, and Image3 contain more than 2 / 3 of the same area. Then Image1 and Image2 can be used as training group 1, Image1 and Image3 as training group 2, and Image2 and Image3 as training group 3. In the second training stage, firstly, the training group 1 (Image1 and Image2) is used to train FeatureNet to obtain the Loss2 of the first training. If the Loss2 of the first training is greater than the preset feature loss threshold, the Loss2 of the first training is used to reversely update the network parameters of FeatureNet; then, the training group 2 (Image1 and Image3) is used to train FeatureNet to obtain the Loss2 of the second training. If the Loss2 of the second training is greater than the preset feature loss threshold, the Loss2 of the second training is used to reversely update the network parameters of FeatureNet; then, the training group 3 (Image2 and Image3) is used to train FeatureNet to obtain the Loss2 of the third training. If the Loss2 of the third training is greater than the preset feature loss threshold, the Loss2 of the third training is used to reversely update the network parameters of FeatureNet; ...; until the Loss2 of the Mth training is less than the preset feature loss threshold, the trained FeatureNet is obtained. Where M is a positive integer.

[0189] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a feature point extraction device, an electronic device, a computer-readable storage medium and corresponding embodiments.

[0190] Fig. 9 It is a structural schematic diagram of a feature point extraction device shown in an embodiment of the present application.

[0191] See also Fig. 9 , the present application provides a feature point extraction device, which may include:

[0192] The image acquisition module 910 is used to acquire the image to be detected and input the image to be detected into a pre-trained feature network model; wherein the feature network model is trained by using the target depth map output by the pre-trained deep network model;

[0193] A first feature information extraction module 920 is used to extract feature information of each pixel in the image to be detected through a feature network model to obtain a target feature map;

[0194] The conversion processing module 930 is used to perform conversion processing on the target feature map to obtain feature description information.

[0195] In one implementation, the conversion processing module 930 may include:

[0196] The first conversion processing submodule is used to detect corner points on the target feature map and convert feature information of the corner points detected in the target feature map into feature description information; or

[0197] The second conversion processing submodule is used to convert the feature information of each pixel in the target feature map into feature description information.

[0198] As can be seen from this example, the solution provided by this application obtains the image to be detected and inputs the image to be detected into a pre-trained feature network model; wherein the feature network model is trained using the target depth map output by the pre-trained deep network model; the feature network model extracts the feature information of each pixel in the image to be detected to obtain the target feature map; the target feature map is converted to obtain feature description information. This application extracts feature points based on the convolutional neural network method, which can deeply optimize scale invariance limitations, sensitivity to perspective changes, occlusion and deformation sensitivity, large-scale scene limitations, computing resource requirements, and limitations on the expressive power of feature descriptors.

[0199] Fig.10 It is a structural diagram of a deep network model training device shown in an embodiment of the present application.

[0200] See also Fig.10 , the deep network model training device of the present application comprises:

[0201] The sample image acquisition module 1010 is used to acquire multiple sample images; wherein the multiple sample images are images captured by a preset camera at different viewing angles;

[0202] The sample image selection module 1020 of this training is used to select two sample images separated by a preset number of frames from multiple sample images;

[0203] A first depth information extraction module 1030 is used to input the two sample images into the deep network model, so as to extract the depth information of each pixel in the two sample images through the deep network model, and obtain two sample depth maps;

[0204] The internal and external parameter acquisition module 1040 is used to obtain the camera internal parameters and camera external parameters of the preset camera;

[0205] A first standard inverse transform operation module 1050 is used to perform a standard inverse transform operation on the two sample depth maps based on the camera intrinsic parameters and the camera extrinsic parameters to obtain two sample mapping maps;

[0206] A deep network model training module 1060 is used to train the deep network model using two sample mapping images to obtain a deep loss value;

[0207] The deep network parameter updating module 1070 is used to adopt the depth loss value to reversely update the network parameters of the deep network model until the depth loss value is less than or equal to the preset depth loss threshold, thereby obtaining a trained deep network model.

[0208] In one embodiment, the deep network model training module 1060 may include:

[0209] A first bilinear interpolation submodule is used to perform bilinear interpolation on two sample images according to two sample mapping images to obtain two predicted images;

[0210] The depth loss value calculation submodule is used to calculate the depth loss value of this training based on two sample images and two prediction images.

[0211] In one embodiment, the two sample images include a first sample image captured based on a first viewing angle and a second sample image captured based on a second viewing angle, and the sample depth map includes a first sample depth map corresponding to the first sample image and a second sample depth map corresponding to the second sample image; the first standard inverse transform operation module 1050 may include:

[0212] A first standard inverse transform operation submodule is used to perform a standard inverse transform operation on the first sample depth map based on the camera intrinsic parameters and the camera extrinsic parameters to obtain a first sample mapping map; and to perform a standard inverse transform operation on the second sample depth map based on the camera intrinsic parameters and the camera extrinsic parameters to obtain a second sample mapping map;

[0213] The first sample mapping diagram is used to represent that each first pixel point in the first sample image is mapped to the first pixel position in the second sample image, and the second sample mapping diagram is used to represent that each second pixel point in the second sample image is mapped to the second pixel position in the first sample image.

[0214] In one embodiment, the first bilinear interpolation submodule may include:

[0215] A sample map pixel position extraction unit, used to extract each first pixel position from the first sample map, and to extract each second pixel position from the second sample map;

[0216] The adjacent pixel point search unit of the sample image is used to search for a plurality of second pixel points adjacent to each first pixel position in the second sample image, and to search for a plurality of first pixel points adjacent to each second pixel position in the first sample image;

[0217] A first bilinear interpolation unit, configured to perform bilinear interpolation on the pixel intensities of the plurality of second pixel points to obtain the pixel intensities of each first pixel position, and to perform bilinear interpolation on the pixel intensities of the plurality of first pixel points to obtain the pixel intensities of each second pixel position;

[0218] The pixel intensity assignment unit of the sample mapping diagram is used to assign the pixel intensity of each first pixel position to the corresponding first pixel point in the first sample mapping diagram to obtain a first predicted image, and to assign the pixel intensity of each second pixel position to the corresponding second pixel point in the second sample mapping diagram to obtain a second predicted image.

[0219] In one embodiment, the depth loss value calculation submodule may include:

[0220] A depth loss value calculation unit for different viewing angles, used to calculate the difference between the first sample image and the first predicted image to obtain the depth loss value of the first viewing angle, and to calculate the difference between the second sample image and the second predicted image to obtain the depth loss value of the second viewing angle, and to calculate the error value of the camera extrinsic parameter;

[0221] The depth loss value calculation unit of this training is used to add the depth loss value of the first perspective, the depth loss value of the second perspective and the error value of the camera extrinsic parameter to obtain the depth loss value of this training.

[0222] Fig.11 It is a structural diagram of a feature network model training device shown in an embodiment of the present application.

[0223] See also Fig.11 , the feature network model training device of the present application comprises:

[0224] The sample image acquisition module 1110 is used to acquire multiple sample images; wherein the multiple sample images are images captured by a preset camera at different viewing angles;

[0225] The sample picture selection module 1120 of this training is used to select two sample pictures separated by a preset number of frames from multiple sample pictures;

[0226] The second depth information extraction module 1130 is used to input the two sample images into a pre-trained deep network model, so as to extract the depth information of each pixel in the two sample images through the deep network model, and obtain two target depth maps;

[0227] The internal and external parameter acquisition module 1140 is used to obtain the camera internal parameters and camera external parameters of the preset camera;

[0228] A second standard inverse transform operation module 1150 is used to perform a standard inverse transform operation on the two target depth maps based on the camera intrinsic parameters and the camera extrinsic parameters to obtain two target mapping maps;

[0229] A feature network model training module 1160 is used to train the feature network model using two target mapping images to obtain a feature loss value;

[0230] The feature network parameter updating module 1170 is used to adopt the feature loss value to reversely update the network parameters of the feature network model until the feature loss value is less than or equal to the preset feature loss threshold, thereby obtaining a trained feature network model.

[0231] In one embodiment, the feature network model training module 1160 may include:

[0232] The second feature information extraction submodule is used to input the two sample images into the feature network model, so as to extract the feature information of each pixel point in the two sample images respectively through the feature network model to obtain two sample feature maps;

[0233] A second bilinear interpolation submodule is used to perform bilinear interpolation on the two sample feature maps according to the two target mapping maps to obtain two prediction feature maps;

[0234] The feature loss value calculation submodule is used to calculate the feature loss value of this training based on two sample feature maps and two predicted feature maps.

[0235] In one embodiment, the two sample images include a first sample image captured based on a first viewing angle and a second sample image captured based on a second viewing angle, and the target depth map includes a first target depth map corresponding to the first sample image and a second target depth map corresponding to the second sample image; the second standard inverse transform operation module 1150 may include:

[0236] A second standard inverse transform operation submodule is used to perform a standard inverse transform operation on the first target depth map based on the camera intrinsic parameters and the camera extrinsic parameters to obtain a first target mapping map; and to perform a standard inverse transform operation on the second target depth map based on the camera intrinsic parameters and the camera extrinsic parameters to obtain a second target mapping map;

[0237] Among them, the first target mapping diagram is used to represent that each first pixel point in the first sample image is mapped to the first pixel position in the second sample image, and the second target mapping diagram is used to represent that each second pixel point in the second sample image is mapped to the second pixel position in the first sample image.

[0238] In one embodiment, the two sample feature maps include a first sample feature map corresponding to the first sample image and a second sample feature map corresponding to the second sample image; the second bilinear interpolation submodule may include:

[0239] A target map pixel position extraction unit, used to extract each first pixel position from the first target map, and to extract each second pixel position from the second target map;

[0240] The adjacent pixel point search unit of the target image is used to search for multiple second pixel points adjacent to each first pixel position in the second sample feature map, and to search for multiple first pixel points adjacent to each second pixel position in the first sample feature map;

[0241] A second bilinear interpolation unit is used to perform bilinear interpolation on the pixel intensities of the plurality of second pixel points to obtain the pixel intensities of each first pixel position, and to perform bilinear interpolation on the pixel intensities of the plurality of first pixel points to obtain the pixel intensities of each second pixel position;

[0242] The pixel intensity assignment unit of the target mapping image is used to assign the pixel intensity of each first pixel position to the corresponding first pixel point in the first target mapping image to obtain a first predicted feature map, and to assign the pixel intensity of each second pixel position to the corresponding second pixel point in the second target mapping image to obtain a second predicted feature map.

[0243] In one embodiment, the feature loss value calculation submodule may include:

[0244] A feature loss value calculation unit for different viewing angles, used to calculate the difference between the first sample feature map and the first predicted feature map to obtain the feature loss value of the first viewing angle, and to calculate the difference between the second sample feature map and the second predicted feature map to obtain the feature loss value of the second viewing angle;

[0245] The feature loss value calculation unit of this training is used to add the feature loss value of the first perspective and the feature loss value of the second perspective to obtain the feature loss value of this training.

[0246] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0247] Fig.12 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0248] See also Fig.12 , the electronic device 1200 includes a memory 1210 and a processor 1220 .

[0249] The processor 1220 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0250] The memory 1210 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. Among them, ROM can store static data or instructions required by the processor 1220 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1210 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1210 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0251] The memory 1210 stores executable codes, and when the executable codes are processed by the processor 1220 , the processor 1220 may execute part or all of the methods described above.

[0252] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0253] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0254] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A characteristic point extraction method, characterized in that: The method comprises: Obtain a picture to be detected, and input the picture to be detected into a pre-trained feature network model; wherein the feature network model is trained using a target depth map output by a pre-trained deep network model; Extracting feature information of each pixel in the image to be detected through the feature network model to obtain a target feature map; The target feature map is converted to obtain feature description information.

2. The method according to claim 1, characterized in that The converting process of the target feature map to obtain feature description information includes: Performing corner point detection on the target feature map, and converting feature information of the corner points detected in the target feature map into feature description information; or, The feature information of each pixel in the target feature map is converted into feature description information.

3. The method according to claim 1, characterized in that: The feature network model is trained in the following way: Acquire multiple sample images; wherein the multiple sample images are images captured by a preset camera at different viewing angles; Selecting two sample pictures separated by a preset number of frames from the multiple sample pictures; Inputting the two sample images into a pre-trained deep network model, so as to respectively extract the depth information of each pixel in the two sample images through the deep network model, and obtain two target depth maps; Obtaining camera intrinsic parameters and camera extrinsic parameters of the preset camera; Based on the camera intrinsic parameters and the camera extrinsic parameters, respectively perform standard inverse transformation operations on the two target depth maps to obtain two target mapping maps; Using the two target maps to train the feature network model to obtain a feature loss value; The feature loss value is used to reversely update the network parameters of the feature network model until the feature loss value is less than or equal to a preset feature loss threshold, thereby obtaining a trained feature network model.

4. The method according to claim 3, characterized in that The adopting the two target mapping images to train the feature network model to obtain a feature loss value includes: Inputting the two sample images into the feature network model, so as to extract feature information of each pixel in the two sample images respectively through the feature network model, and obtain two sample feature maps; According to the two target mapping maps, bilinear interpolation is performed on the two sample feature maps respectively to obtain two prediction feature maps; The feature loss value of this training is calculated based on the two sample feature maps and the two predicted feature maps.

5. The method according to claim 4, characterized in that The two sample images include a first sample image acquired based on a first viewing angle and a second sample image acquired based on a second viewing angle, and the target depth map includes a first target depth map corresponding to the first sample image and a second target depth map corresponding to the second sample image; The method of performing a standard inverse transformation operation on the two target depth maps based on the camera intrinsic parameters and the camera extrinsic parameters to obtain two target mapping maps includes: Based on the camera intrinsic parameters and the camera extrinsic parameters, a standard inverse transform operation is performed on the first target depth map to obtain a first target mapping map; and based on the camera intrinsic parameters and the camera extrinsic parameters, a standard inverse transform operation is performed on the second target depth map to obtain a second target mapping map; Among them, the first target mapping map is used to represent that each first pixel point in the first sample image is respectively mapped to the first pixel position in the second sample image, and the second target mapping map is used to represent that each second pixel point in the second sample image is respectively mapped to the second pixel position in the first sample image.

6. The method according to claim 5, characterized in that The two sample feature maps include a first sample feature map corresponding to the first sample image and a second sample feature map corresponding to the second sample image; and the two sample feature maps are bilinearly interpolated according to the two target mapping maps to obtain two predicted feature maps, including: extracting each first pixel position from the first object map, and extracting each second pixel position from the second object map; Searching for a plurality of second pixel points adjacent to each of the first pixel positions in the second sample feature map, and searching for a plurality of first pixel points adjacent to each of the second pixel positions in the first sample feature map; Performing bilinear interpolation on the pixel intensities of the plurality of second pixel points to obtain the pixel intensities of the respective first pixel positions, and performing bilinear interpolation on the pixel intensities of the plurality of first pixel points to obtain the pixel intensities of the respective second pixel positions; The pixel intensities of the first pixel positions are assigned to the corresponding first pixel points in the first target mapping image to obtain a first predicted feature map, and the pixel intensities of the second pixel positions are assigned to the corresponding second pixel points in the second target mapping image to obtain a second predicted feature map.

7. The method according to claim 6, characterized in that The calculating the feature loss value of this training according to the two sample feature maps and the two predicted feature maps includes: Calculating the difference between the first sample feature map and the first predicted feature map to obtain a feature loss value of the first viewing angle, and calculating the difference between the second sample feature map and the second predicted feature map to obtain a feature loss value of the second viewing angle; The feature loss value of the first perspective is added to the feature loss value of the second perspective to obtain the feature loss value of this training.

8. The method according to claim 5, characterized in that The deep network model is trained in the following way: Inputting the two sample images into the deep network model, so as to respectively extract the depth information of each pixel in the two sample images through the deep network model, and obtain two sample depth maps; Based on the camera intrinsic parameters and the camera extrinsic parameters, respectively perform standard inverse transformation operations on the two sample depth maps to obtain two sample mapping maps; Using the two sample maps to train the deep network model to obtain a depth loss value; The depth loss value is used to reversely update the network parameters of the deep network model until the depth loss value is less than or equal to a preset depth loss threshold, thereby obtaining a trained deep network model.

9. The method according to claim 8, characterized in that The using the two sample mapping images to train the deep network model to obtain a depth loss value includes: According to the two sample mapping images, bilinear interpolation is performed on the two sample images respectively to obtain two predicted images; According to the two sample images and the two predicted images, the depth loss value of this training is calculated.

10. The method according to claim 9, characterized in that The two sample depth maps include a first sample depth map corresponding to the first sample picture and a second sample depth map corresponding to the second sample picture, the two sample mapping maps include a first sample mapping map corresponding to the first sample depth map and a second sample mapping map corresponding to the second sample depth map, and the two predicted pictures include a first predicted picture corresponding to the first sample mapping map and a second predicted picture corresponding to the second sample mapping map; The calculating the depth loss value of this training according to the two sample images and the two predicted images includes: Calculating the difference between the first sample image and the first predicted image to obtain the depth loss value of the first viewing angle, calculating the difference between the second sample image and the second predicted image to obtain the depth loss value of the second viewing angle, and calculating the error value of the camera extrinsic parameter; The depth loss value of the first perspective, the depth loss value of the second perspective, and the error value of the camera extrinsic parameter are added to obtain the depth loss value of this training.

11. A characteristic point extraction device, characterized in that: The device comprises: The module for acquiring the image to be detected is used to acquire the image to be detected and input the image to be detected into a pre-trained feature network model; wherein the feature network model is trained by using the target depth map output by the pre-trained deep network model; A first feature information extraction module is used to extract feature information of each pixel in the image to be detected through the feature network model to obtain a target feature map; The conversion processing module is used to convert the target feature map to obtain feature description information.

12. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 10.

Images