Method and device for extracting footprint, footprint extraction equipment and storage medium

Abstract

The embodiment of the invention discloses a footprint extraction method and device, footprint extraction equipment and a storage medium. The method comprises the following steps: acquiring footprint images of a to-be-extracted footprint in a plurality of preset directions; performing feature extraction on each footprint image to obtain feature information of each footprint image; and performing stereoscopic model reconstruction according to each feature message to obtain a three-dimensional model of the footprint to be extracted. In this way, compared with footprint extraction through gypsum molding, photographing can store the features of the footprint to be extracted, the footprint to be extracted is not prone to being damaged, and therefore accurate extraction of the footprint to be extracted can be achieved.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and particularly relates to a method and device for footprint extraction, a footprint extraction device, and a storage medium. Background Art

[0002] Currently, based on whether the bearing can undergo plastic deformation, the footprints left by a person walking on the bearing can be divided into planar footprints and three-dimensional footprints. Compared with planar footprints, three-dimensional footprints can not only reflect the pattern image features of the bottom of the footprint, but also reflect the three-dimensional shape of the bottom of the footprint and the dynamic shape features of human walking movement, playing a very important role in determining the nature of the case, narrowing the scope of investigation, providing case linkage, analyzing the characteristics of the perpetrator, and reconstructing the crime scene. Therefore, the extraction of three-dimensional footprints plays an important role in criminal technology and other fields.

[0003] In the prior art, professional personnel are required to make a plaster mold of the footprint to be extracted to achieve the extraction of three-dimensional footprints. However, when the bearing is a soft object such as snow or mud, it is easy to damage the three-dimensional footprint during the plaster molding process, and thus the three-dimensional footprint cannot be extracted completely and accurately. Summary of the Invention

[0004] To solve the above technical problems, embodiments of the present application provide a method and device for footprint extraction, a footprint extraction device, and a storage medium, which can improve the accuracy of footprint extraction.

[0005] According to one aspect of the embodiments of the present application, a method for footprint extraction is provided, including: obtaining footprint images of the footprint to be extracted in a plurality of preset directions; performing feature extraction on each of the footprint images to obtain feature information of each of the footprint images; and performing three-dimensional model reconstruction based on each of the feature messages to obtain a three-dimensional model of the footprint to be extracted.

[0006] In some embodiments, it is applied to a footprint extraction device; the footprint extraction device includes a camera; the obtaining footprint images of the footprint to be extracted in a plurality of preset directions includes: when the shooting angle of the camera is perpendicular to the footprint to be extracted, obtaining a first image in a direction perpendicular to the footprint to be extracted; adjusting the shooting angle of the camera to a preset angle; based on the shooting angle, controlling the camera to circularly shoot the footprint to be extracted to obtain a plurality of second images in the direction corresponding to the preset angle; and determining the first image and the second images as the footprint images.

[0007] In some embodiments, reconstructing a stereo model according to each of the feature messages to obtain a three-dimensional model of the footprint to be extracted includes: performing sparse reconstruction according to the feature messages to obtain a reconstruction result; obtaining reconstruction parameters according to the reconstruction result; inputting the reconstruction parameters into a preset depth map estimation model to obtain a depth map corresponding to the footprint to be extracted; and performing three-dimensional reconstruction based on the depth map to obtain the three-dimensional model.

[0008] In some embodiments, performing sparse reconstruction according to the feature messages to obtain a reconstruction result includes: establishing a matching relationship between each of the footprint images to obtain a plurality of matching image pairs; performing feature point matching on each of the matching image pairs according to the feature messages to determine matching points corresponding to each of the matching image pairs; and performing sparse reconstruction based on each of the matching points to obtain the reconstruction result.

[0009] In some embodiments, it is applied to a footprint extraction device; the footprint extraction device includes a camera; the reconstruction result includes sparse point information; obtaining reconstruction parameters according to the reconstruction result includes: obtaining camera parameters of the camera and image information of each of the footprint images; obtaining an external camera parameter according to the image information; obtaining a depth range according to the external parameter matrix and the sparse point information; obtaining a similarity score for each of the matching image pairs, and selecting the matching image pair with the highest similarity based on the similarity score among each of the matching image pairs; obtaining sparse point information of the selected matching image pair to obtain reference sparse point information; and determining the camera internal parameter, the external camera parameter, the depth range, and the reference sparse point information as the reconstruction parameters.

[0010] In some embodiments, obtaining the similarity score for each of the matching image pairs includes: obtaining intersection sparse points corresponding to each of the matching image pairs based on the sparse point information; for each matching image pair, calculating an angle between each of the corresponding intersection sparse points and the camera center corresponding to the camera; calculating score contributions of each of the intersection sparse points according to each of the angles; and determining the sum of the score contributions of each of the intersection sparse points as the similarity score of the matching image pair.

[0011] In some embodiments, performing three-dimensional reconstruction based on the depth map to obtain the three-dimensional model includes: obtaining vertex coordinates corresponding to the footprint to be extracted and color information corresponding to the vertex coordinates based on the depth map; and integrating the vertex coordinates and the color information to obtain the three-dimensional model.

[0012] According to one aspect of the embodiments of the present application, there is provided a device for footprint extraction, including: an image acquisition module configured to acquire footprint images of the footprint to be extracted in a plurality of preset directions; a feature extraction module configured to perform feature extraction on each of the footprint images to obtain feature information of each of the footprint images; and a reconstruction module configured to perform three-dimensional model reconstruction based on each of the feature messages to obtain a three-dimensional model of the footprint to be extracted.

[0013] According to one aspect of the embodiments of the present application, there is provided a footprint extraction device, including: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the footprint extraction device to implement the above-mentioned method for footprint extraction.

[0014] According to one aspect of the embodiments of the present application, there is provided a computer-readable storage medium having stored thereon computer-readable instructions, which, when executed by a processor of a computer, cause the computer to execute the above-mentioned method for footprint extraction.

[0015] In the technical solution provided by the embodiments of the present application, by acquiring footprint images of the footprint to be extracted in a plurality of preset directions, then performing feature extraction on each footprint image to obtain feature information of each footprint image, and then performing three-dimensional model reconstruction based on each feature message to obtain a three-dimensional model of the footprint to be extracted. In this way, three-dimensional modeling is performed on the images collected by taking pictures to realize the extraction of the footprint to be extracted. Compared with extracting the footprint by plaster molding, taking pictures can preserve the features of the footprint to be extracted and is not easy to damage the footprint to be extracted, thereby enabling the accurate extraction of the footprint to be extracted.

[0016] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. Description of the Drawings

[0017] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts. In the drawings:

[0018] Figure 1 is a schematic diagram of a scenario for footprint extraction shown in an exemplary embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a footprint extraction device shown in an exemplary embodiment of the present application;

[0020] Figure 3 is a flowchart of a method for extracting footprints shown in an exemplary embodiment of the present application;

[0021] Figure 4 is a flowchart of a method for extracting footprints shown in another exemplary embodiment of the present application;

[0022] Figure 5 is a block diagram of a device for extracting footprints shown in an exemplary embodiment of the present application;

[0023] Figure 6 shows a schematic structural diagram of a footprint extraction device suitable for implementing the embodiments of the present application. Detailed implementation manners

[0024] Here, an exemplary embodiment will be described in detail, and its examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the drawings are only exemplary descriptions and do not necessarily include all contents and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0027] In the present application, "a plurality of" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0028] Please refer to Figure 1 , Figure 1 is a schematic diagram of a scenario for extracting footprints shown in an exemplary embodiment of the present application.

[0029] As Figure 1As shown, when a person walks and stands on the bearing body 11, footprints 12 of the person will be left on the bearing body 11. When it is necessary to extract the footprints, the footprint extraction device 13 is used to extract the footprints 12.

[0030] It should be noted that footprints are divided into planar footprints and three-dimensional footprints.

[0031] Planar footprints are usually formed on relatively hard and smooth bearing bodies, such as cement floors, ceramic tile floors, etc. When a person walks or stands on these bearing bodies, since the ground is hard and no depression can be formed, the footprints formed are planar footprints. Planar footprints have two-dimensional structural characteristics, that is, they can only reflect the image of the raised parts of the sole, but cannot present a complete three-dimensional form.

[0032] Three-dimensional footprints are formed on relatively soft ground or bearing bodies with strong plasticity, such as soil, sand, etc. When a person walks or stands on these bearing bodies, due to the softness of the ground, the footprints will present a sunken shape and have three-dimensional characteristics. The concave and convex characteristics of three-dimensional footprints are opposite to those on the human foot, and can clearly reflect the concave and convex characteristics of the sole and side of the foot, shoe or sock.

[0033] Compared with planar footprints, three-dimensional footprints can not only reflect the pattern image characteristics of the footprint bottom, but also reflect the three-dimensional form of the footprint bottom and the dynamic form characteristics of human walking movement, and play a very important role in determining the nature of the case, narrowing the scope of investigation, providing case linking, analyzing the characteristics of the perpetrator, and conducting on-site reconstruction, etc.

[0034] The footprints mentioned in this embodiment are all three-dimensional footprints.

[0035] Please refer to Figure 2 , Figure 2 which is a schematic diagram of a footprint extraction device shown in an exemplary embodiment of the present application.

[0036] As Figure 2 shown, the footprint extraction device 21 includes a camera 22, a pan-tilt 23, a robotic arm 24, a servo 25, an ultrasonic measuring instrument 26, and an adjustable bracket 27.

[0037] Specifically, the camera 22 is used to take pictures of the footprints.

[0038] The camera 22 is connected to the robotic arm 24.

[0039] An ultrasonic measuring instrument 26 is arranged beside the camera 22. The ultrasonic measuring instrument 26 is used to detect the distance between the camera and the footprint.

[0040] The robotic arm can be a multi-axis robotic arm, for example: a four-axis robotic arm, a five-axis robotic arm, a six-axis robotic arm or a seven-axis robotic arm, etc., which is not limited here.

[0041] The robotic arm 24 is used to move the camera 22. The robotic arm 24 is also used to adjust the shooting direction of the camera 22.

[0042] Above the robotic arm 24, there is a pan-tilt head 23 connected. The pan-tilt head 23 can support the robotic arm 24. The pan-tilt head 23 includes multiple rotating axes, which are used to smoothly move the camera in multiple directions and improve the stability of the camera.

[0043] Combining the robotic arm and the pan-tilt head can accurately and omni-directionally control the shooting position and shooting angle of the camera, so as to complete the image acquisition of footprints from multiple angles.

[0044] Above the pan-tilt head 23, there is a servo 25 placed. The servo 25 can be electrically connected to the pan-tilt head 23; it is used to control the rotation of the pan-tilt head.

[0045] Below the pan-tilt head 23, there are multiple adjustable brackets 27 provided. The length of the adjustable brackets is greater than the length of the robotic arm 24. The number of adjustable brackets can be 3 or more, and the lengths of the adjustable brackets can be placed separately. In this way, even if the bearing body is unstable ground, such as: snow, mud, etc., the footprint extraction device can be stably placed on the bearing body.

[0046] See Figure 3 , Figure 3 which is a flowchart of a method for footprint extraction shown in an exemplary embodiment of the present application.

[0047] As Figure 3 shown, in an exemplary embodiment, the method for footprint extraction at least includes step S310 to step S330, which are introduced in detail as follows:

[0048] Step S310, obtain footprint images of the footprint to be extracted in multiple preset directions.

[0049] It should be noted that the footprint to be extracted is a three-dimensional footprint. The footprint image is an image including this three-dimensional footprint.

[0050] The footprint image is a color image with a resolution size of the preset resolution.

[0051] Further, obtaining footprint images of the footprint to be extracted in multiple preset directions includes: when the shooting angle of the camera is perpendicular to the footprint to be extracted, obtaining a first image perpendicular to the direction of the footprint to be extracted; adjusting the shooting angle of the camera to a preset angle; based on the shooting angle, controlling the camera to circularly shoot the footprint to be extracted to obtain a plurality of second images in the direction corresponding to the preset angle; and determining the first image and the second images as footprint images. In this way, by adjusting the shooting angle of the camera, a plurality of footprint images in multiple directions are obtained. A large number of images with different inclination angles from the plane where the footprint is located and equidistant from the center of the footprint are collected, so that the footprint images can include the features of the footprint to be extracted from various angles, and further the accuracy of the footprint extracted based on the footprint images is higher.

[0052] It should be noted that the preset angle can include a preset angle or multiple preset angles. In some embodiments, the more the number of preset angles, the higher the accuracy of constructing the footprint to be extracted.

[0053] The preset angle can include one or more of 15 degrees, 30 degrees, and 45 degrees.

[0054] The shooting angle of the camera is the angle between the plane corresponding to the footprint to be extracted and the shooting direction of the camera when there is a preset distance between the perpendicular lines from the camera to the center point of the footprint to be extracted.

[0055] In some embodiments, the preset angles include 15 degrees, 30 degrees, and 45 degrees. The preset distance is 20 cm. For each shooting angle, the number of second images taken is 20.

[0056] The footprint extraction device is erected through an adjustable bracket so that the camera is directly above the center point of the three-dimensional footprint, and then the shooting angle of the camera is controlled to be perpendicular to the footprint to be extracted, that is, vertically downward, and then the camera is controlled to shoot the footprint to be extracted to obtain a first image.

[0057] Then, a servo is used to control the robotic arm and the pan-tilt to make the shooting direction of the camera form a 45° angle with the plane corresponding to the footprint to be extracted, and an ultrasonic measuring instrument is used to calibrate the position of the camera so that there is a 20 cm interval between the perpendicular lines from the camera to the center point of the footprint to be extracted, thereby controlling the shooting angle of the camera to be 45 degrees. Then, based on this 45-degree angle, the camera is controlled to circularly shoot the footprint to be extracted to obtain 20 second images in the direction corresponding to the preset angle, that is, the camera is controlled to move and shoot on the plane corresponding to the 45-degree shooting angle of the camera along a circle with a radius of 20 cm, which is the interval between the perpendicular lines from the camera to the center point of the footprint to be extracted, to obtain 20 second images corresponding to 45 degrees.

[0058] Use a servo motor to control the robotic arm and the pan-tilt head to make the shooting direction of the camera form a 30° angle with the plane corresponding to the footprint to be extracted, and calibrate the position of the camera through an ultrasonic measuring instrument so that the distance between the perpendicular lines from the camera to the center point of the footprint to be extracted is 20 cm, thereby controlling the shooting angle of the camera to be 30 degrees. Then, based on this 30 degrees, control the camera to circularly shoot the footprint to be extracted, and obtain 20 second images in the directions corresponding to the preset angles, that is, control the camera to move and shoot on the plane corresponding to the shooting angle of 30 degrees of the camera, on a circle with a radius of 20 cm between the perpendicular lines from the camera to the center point of the footprint to be extracted, and obtain 20 second images corresponding to 30 degrees.

[0059] Use a servo motor to control the robotic arm and the pan-tilt head to make the shooting direction of the camera form a 15° angle with the plane corresponding to the footprint to be extracted, and calibrate the position of the camera through an ultrasonic measuring instrument so that the distance between the perpendicular lines from the camera to the center point of the footprint to be extracted is 20 cm, thereby controlling the shooting angle of the camera to be 15 degrees. Then, based on this 15 degrees, control the camera to circularly shoot the footprint to be extracted, and obtain 20 second images in the directions corresponding to the preset angles, that is, control the camera to move and shoot on the plane corresponding to the shooting angle of 15 degrees of the camera, on a circle with a radius of 20 cm between the perpendicular lines from the camera to the center point of the footprint to be extracted, and obtain 20 second images corresponding to 15 degrees.

[0060] Then, determine the first image, the 20 second images corresponding to 45 degrees, the 20 second images corresponding to 30 degrees, and the 20 second images corresponding to 15 degrees as footprint images.

[0061] It should be noted that after obtaining the footprint images of the footprint to be extracted in multiple preset directions, the robotic arm and the pan-tilt head can be controlled to return to the default initial position, so as to facilitate the extraction of the next footprint and the storage of the footprint extraction device.

[0062] Step S320: Extract features from each footprint image to obtain the feature information of each footprint image.

[0063] It should be noted that before extracting features from each footprint image, it may also include: performing super-resolution processing on each footprint feature.

[0064] Furthermore, performing super-resolution processing on each footprint feature includes: sequentially performing preprocessing, enhancement processing, and image post-processing on each footprint image.

[0065] Specifically, preprocess each footprint image, including: obtaining the image data of each footprint image separately. Convert the format of this image data. Then, pre-fill and modulo-fill the format-converted image data according to the scaling ratio of the image, so that the size of the image corresponding to the filled image data is the preset image size.

[0066] Specifically, use the preset Real-ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) algorithm to pre-fill and modulo-fill the format-converted image data according to the scaling ratio of the image.

[0067] It should be noted that the image data is NumPy array data of (height, width, channels). The format-converted image data is PyTorch tensor data of (channels, height, width). The type of the values in the PyTorch tensor data is floating-point type.

[0068] Furthermore, perform enhancement processing on each footprint image, including: inputting the image data of each preprocessed footprint image into a preset image enhancement model to perform image enhancement on each footprint image.

[0069] Furthermore, perform post-processing on each footprint image, including: removing the data filled in the preprocessing and enhancement processing from the image data; obtaining the bit depth of the image. Obtain the scaling range corresponding to this bit depth; scale the image data according to this scaling range; convert the image data into NumPy array data of (height, width, channels).

[0070] It should be noted that the enhancement processing may also fill the image data, then removing the data filled in the preprocessing and enhancement processing from the image data includes: the pre-filled image data, the modulo-filled image data, and the image data filled in the enhancement processing.

[0071] The bit depth of an image refers to the number of bits used by each pixel in the image. These bits are used to represent the color or gray level of the pixel. The higher the bit depth, the more color or gray levels each pixel can represent, and thus the higher the quality and details of the image.

[0072] In this embodiment, the bit depth of the image includes 8 bits or 16 bits. The scaling range corresponding to the 8-bit bit depth is the preset first reference range; the scaling range corresponding to the 16-bit bit depth is the preset second reference range.

[0073] Further, feature extraction is performed on each footprint image to obtain the feature information of each footprint image, including: using a preset SIFT (Scale-Invariant Feature Transform) feature extraction algorithm to perform feature extraction on each footprint image to obtain the feature information of each footprint image.

[0074] It should be noted that corresponding feature information can be extracted from each image after feature extraction. Specifically, the feature information includes: the number of feature points extracted, the coordinates of the extracted feature points, the image scale, the gradient direction of the extracted feature points, and the feature vectors corresponding to each feature point. The feature vector is a 128-dimensional vector used to represent the feature descriptor of the feature point, which represents the gradient direction information of the image area around the feature point.

[0075] Further, after obtaining the feature information of each footprint image, it may further include: storing the feature information. Specifically, the feature information is stored in a.db file.

[0076] Step S330, perform stereo model reconstruction based on each feature message to obtain a three-dimensional model of the footprint to be extracted.

[0077] In this embodiment, by obtaining the footprint images of the footprint to be extracted in multiple preset directions, then performing feature extraction on each footprint image to obtain the feature information of each footprint image, and then performing stereo model reconstruction based on each feature message to obtain a three-dimensional model of the footprint to be extracted. In this way, three-dimensional modeling is performed on the images collected by photographing to achieve the extraction of the footprint to be extracted. Compared with extracting the footprint by plaster casting, photographing can preserve the features of the footprint to be extracted and is not easy to damage the footprint to be extracted, so as to accurately extract the footprint to be extracted.

[0078] Further, performing stereo model reconstruction based on each feature message to obtain a three-dimensional model of the footprint to be extracted includes: performing sparse reconstruction according to the feature message to obtain a reconstruction result; obtaining reconstruction parameters according to the reconstruction result; inputting the reconstruction parameters into a preset depth map estimation model to obtain a depth map corresponding to the footprint to be extracted; and performing three-dimensional reconstruction based on the depth map to obtain a three-dimensional model. In this way, sparse reconstruction is first performed, and three-dimensional reconstruction is performed based on the reconstruction result of the sparse reconstruction, reducing the amount of data for three-dimensional reconstruction, thereby improving the rate of three-dimensional reconstruction.

[0079] It should be noted that the feature messages of the footprint images of different footprints to be extracted in multiple preset directions can comprehensively and accurately represent the three-dimensional features of the footprint to be extracted. Then, using this feature message, a stereo model of the footprint to be extracted can be reconstructed, that is, three-dimensional modeling, so that the modeled model more conforms to the three-dimensional features of the footprint to be extracted, improving the accuracy of modeling.

[0080] Further, perform sparse reconstruction based on the feature messages to obtain a reconstruction result, including: establishing a matching relationship between each footprint image to obtain multiple pairs of matching images; performing feature point matching on each pair of matching images according to the feature messages to determine the matching points corresponding to each pair of matching images; performing sparse reconstruction based on each matching point to obtain a reconstruction result. In this way, by performing feature point matching on the matching pairs of matching images, determining the matching points corresponding to each pair of matching images, and then performing sparse reconstruction based on each matching point to obtain a reconstruction result, the sparse reconstruction of the footprint to be extracted is realized. At the same time, performing sparse reconstruction based on the matching points reduces the data volume of subsequent 3D reconstruction and improves the rate of 3D reconstruction.

[0081] Further, establishing a matching relationship between each footprint image to obtain multiple pairs of matching images, including: using a preset exhaustive matching algorithm to establish a matching relationship between each footprint image to obtain multiple pairs of matching images.

[0082] Further, performing feature point matching on each pair of matching images according to the feature messages to determine the matching points corresponding to each pair of matching images, including: for each pair of matching images, taking one footprint image as the target image and the other footprint image as the image to be matched; using a preset KD-tree algorithm to establish a search tree according to the feature messages corresponding to the image to be matched; the nodes of the search tree are the feature vectors corresponding to the feature points of the image to be matched; sequentially determining the similarity between the feature vectors corresponding to the nodes in the search tree and the feature points in the target image according to the search tree to determine the matching points corresponding to this pair of matching images.

[0083] It should be noted that after determining the matching points corresponding to each pair of matching images, it further includes: filtering the matching points using a preset geometric verification algorithm.

[0084] Further, performing sparse reconstruction based on each matching point to obtain a reconstruction result, including: selecting one pair of matching images from each pair of matching images; taking the pose of one footprint image in the selected pair of matching images as the unit matrix; obtaining the essential matrix corresponding to the selected pair of matching images through the matching points of the selected pair of matching images; decomposing the essential matrix to obtain the pose of the other footprint image in the selected pair of matching images; triangulating the poses of the two footprint images in the pair of matching images to generate alternative sparse points; performing incremental reconstruction on the alternative sparse points using other pairs of matching images to obtain multiple sparse points; extracting the sparse point information of each sparse point; determining the sparse point information of each sparse point as the reconstruction result.

[0085] It should be noted that the common triangulation methods include linear triangulation and geometric triangulation. Among them, linear triangulation uses the two matching image coordinates, image poses, and the camera's internal parameter matrix to obtain the 3D point coordinates of the matching point.

[0086] Further, incremental reconstruction is performed on the alternative sparse points using other matching image pairs to obtain sparse points, including: sequentially selecting the matching image pair with the largest number of matching point pairs from other matching image pairs; and after each selection, taking the pose of one footprint image in the selected matching image pair as the identity matrix; obtaining the essential matrix corresponding to the selected matching image pair through the matching points of the selected matching image pair; decomposing the essential matrix to obtain the pose of the other footprint image in the selected matching image pair; triangulating the poses of the two footprint images in the matching image pair to generate alternative sparse points again. Optimize all the generated alternative sparse points to obtain sparse points.

[0087] Further, optimize all the generated alternative sparse points to obtain sparse points, including: optimizing all the generated alternative sparse points using the preset bundle adjustment method to obtain sparse points.

[0088] It should be noted that the sparse point information includes: the ID message of each sparse point, the coordinates of each sparse point, the color message of each sparse point, the error message of the sparse point, the sparse depth value of the sparse point, and the index message representing each sparse point in each footprint image.

[0089] Among them, the error message of the sparse point is the message used to represent the error of the pixel point in the sparse reconstruction process.

[0090] Further, the reconstruction result includes sparse point information; obtaining reconstruction parameters according to the reconstruction result, including: obtaining the camera parameters of the camera and the image information of each footprint image; obtaining the extrinsic parameters of the camera according to the image information; obtaining the depth range according to the extrinsic parameter matrix and the sparse point information; obtaining the similarity score of each matching image pair, and selecting the matching image pair with the highest similarity from each matching image pair based on the similarity score; obtaining the sparse point information of the selected matching image pair to obtain the reference sparse point information; determining the camera intrinsic parameters, camera extrinsic parameters, depth range, and reference sparse point information as the reconstruction parameters. In this way, the reference sparse point information of the matching image pair with the highest similarity, the camera intrinsic parameters of the camera, the camera extrinsic parameters, and the depth range can be determined as the reconstruction parameters, so as to facilitate three-dimensional reconstruction using the reconstruction parameters and realize the extraction of the footprint to be extracted.

[0091] It should be noted that the camera parameters of the camera include the camera ID of the camera, the camera attributes of the camera, the image size captured by the camera, and the camera intrinsic parameters of the camera.

[0092] The image information of the footprint image includes: the image ID of the footprint image, the quaternion of the footprint image, the rotation vector of the footprint image; the camera ID of the camera that captured the footprint image, and the name of the footprint image.

[0093] In some embodiments, the quaternion of the footprint image, i.e., the translation vector corresponding to the footprint image, represents the translation amount of the projection of each pixel point in the footprint image from the world coordinate system to the camera coordinate system.

[0094] The rotation vector of the footprint image represents the rotation amount of the projection of each pixel point in the footprint image from the world coordinate system to the camera coordinate system;

[0095] Combining the rotation vector of the footprint image and the translation vector corresponding to the footprint image can represent the reconstructed pose of the footprint image.

[0096] In some embodiments, the camera parameters of the camera, the image information of each footprint image, the reconstruction result, and a preset configuration file can be stored in a preset output folder, e.g., the sparse folder.

[0097] It should be noted that the extrinsic camera parameters are parameters used to describe the position and pose of the camera in the world coordinate system. That is, for each footprint image, there is a corresponding extrinsic camera parameter. The extrinsic camera parameter is a 4×4 matrix.

[0098] Furthermore, obtaining the extrinsic camera parameters according to the image information includes: converting the quaternion of each footprint image into a rotation matrix; obtaining the extrinsic camera parameters corresponding to each footprint image according to the rotation matrix corresponding to each footprint image and the translation vector of each footprint image respectively.

[0099] Furthermore, converting the quaternion of each footprint image into a rotation matrix includes: calculating to obtain the rotation matrix corresponding to the i-th footprint image. Wherein, R i is the rotation matrix corresponding to the i-th footprint image; w i is the first data of the quaternion of the i-th footprint image; x i is the second data of the quaternion of the i-th footprint image; y i is the third data of the quaternion of the i-th footprint image; z i is the fourth data of the quaternion of the i-th footprint image.

[0100] Specifically, for the i-th footprint image, its quaternion is (w i , x i , y i , z i ). Convert this quaternion into a 3×3 rotation matrix. Assign the rotation matrix to the first 3×3 elements of the extrinsic parameter matrix; assign the translation vector (Tx i , Ty i , Tz i ) of the i-th footprint image to the first 3 elements of the fourth column of the extrinsic parameter matrix; set the fourth element in the fourth row of the extrinsic parameter matrix to 1 to obtain the extrinsic camera parameter corresponding to the i-th footprint image.

[0101] Further, obtaining the depth range according to the external parameter matrix and the sparse point information includes: multiplying the coordinates of each sparse point corresponding to the i-th footprint image by the external camera parameters corresponding to the i-th footprint image to obtain an alternative matrix; sorting the values of each element of the alternative matrix from smallest to largest to obtain a depth sequence. Determining the value of the element at the 1% position in the depth sequence as the lower limit value of the depth range; determining the value of the element at the 99% position in the depth sequence as the upper limit value of the depth range.

[0102] Multiplying the i-th footprint image by the external camera parameters corresponding to the i-th footprint image can transform each sparse point from the world coordinate system to the camera coordinate system.

[0103] Further, obtaining the similarity score of each pair of matching images includes: obtaining the intersection sparse points corresponding to each pair of matching images based on the sparse point information; for each pair of matching images, calculating the angle between each corresponding intersection sparse point and the camera center corresponding to the camera; calculating the score contribution of each intersection sparse point according to each angle; determining the sum of the score contributions of each intersection sparse point as the similarity score of the pair of matching images. In this way, the score contribution of each intersection sparse point is determined by the angle between the intersection sparse point corresponding to the pair of matching images and the camera center corresponding to the camera, and the sum of the score contributions of each intersection sparse point is determined as the similarity score of the pair of matching images, so as to select the pair of matching images with the highest similarity from the pairs of matching images and use the reference sparse point information of the selected pair of matching images for three-dimensional reconstruction, further reducing the data volume of subsequent three-dimensional reconstruction and improving the rate of three-dimensional reconstruction.

[0104] The intersection sparse points corresponding to the pair of matching images are the sparse points corresponding to both footprint images in the pair of matching images.

[0105] Specifically, all sparse points of the two footprint images in the pair of matching images can be determined respectively by the index information of each sparse point in each footprint image. Then, the sparse points corresponding to both footprint images are determined by the coordinates of all sparse points of the two footprint images, and the intersection sparse points corresponding to the pair of matching images are obtained.

[0106] Specifically, the pair of matching images includes the i-th footprint image and the j-th footprint image; for the i-th footprint image, Geometry I i is the set of sparse points of the i-th footprint image; for the j-th footprint image, Geometry I j is the set of sparse points of the j-th footprint image. Then I 交 = I i ∩ I j and I 交 is the set of intersection sparse points corresponding to the pair of matching images.

[0107] Further, for each pair of matching images, calculate the angles between the corresponding sparse intersection points and the camera center corresponding to the camera, including: obtaining the position vectors corresponding to each footprint image in the pair of matching images; obtaining the spatial position vectors corresponding to each sparse intersection point in the pair of matching images; and obtaining the angles between each sparse intersection point and the camera center corresponding to the camera according to the position vectors corresponding to each footprint image and the spatial position vectors corresponding to each sparse intersection point in the pair of matching images.

[0108] It should be noted that the position vectors corresponding to each footprint image and the spatial position vectors corresponding to each sparse intersection point in the pair of matching images can be obtained through image information and camera parameters.

[0109] Further, obtaining the angles between each sparse intersection point and the camera center corresponding to the camera according to the position vectors corresponding to each footprint image and the spatial position vectors corresponding to each sparse intersection point in the pair of matching images includes: by calculating obtain the angle between the k-th sparse intersection point and the camera center corresponding to the camera. Wherein, θ k is the angle between the k-th sparse intersection point and the camera center corresponding to the camera; C i is the position vector corresponding to the i-th footprint image; C j is the position vector corresponding to the j-th footprint image; P k is the spatial position vector corresponding to the k-th sparse intersection point; "||||" represents taking the norm.

[0110] Further, calculate the score contributions of each sparse intersection point according to the respective angles, including: by calculating obtain the score contribution of the k-th sparse intersection point. Wherein, S k is the score contribution of the k-th sparse intersection point; θ0 is a preset angle parameter; σ is a standard parameter.

[0111] It should be noted that when θ k ≤θ0, σ is a preset first standard parameter σ1; when θ k >θ0, σ is a preset second standard parameter σ2; σ1 < σ2. In some embodiments, σ1 is 1; σ2 is 10.

[0112] Further, determine the sum of the score contributions of each sparse intersection point as the similarity score of the pair of matching images, including: by calculating obtain the similarity score of the pair of matching images. Wherein, S is the similarity score of the pair of matching images; I 交 is the sparse intersection point of the pair of matching images; k represents the k-th sparse intersection point.

[0113] Further, input the reconstruction parameters into a preset depth map estimation model to obtain the depth map corresponding to the footprint to be extracted, that is, input the camera internal parameters, camera external parameters, depth range, and reference sparse point information into the preset depth map estimation model to obtain the depth map corresponding to the footprint to be extracted.

[0114] It should be noted that the preset depth map estimation model is a learning-based ColmapPatch model.

[0115] The preset depth map estimation model includes a triangulation module, an adaptive propagation module, and an adaptive matching cost module. Among them, the triangulation module is used to obtain the depth map according to the reference sparse point information. The adaptive propagation module is used to capture the spatio-temporal information and dependency relationships of each depth value in the depth map, thereby improving the performance and accuracy of the model. The adaptive matching cost module is used to calculate the similarity between pixel points. In some embodiments, one depth value corresponds to one pixel point.

[0116] Further, the triangulation module obtains the depth map through the following method: triangulate the reference sparse points in the reference sparse point information to determine the vertex sparse points of multiple triangles; convert the coordinates of each vertex sparse point to the camera coordinates; obtain the normal vectors of the planes corresponding to each triangle according to each camera coordinate; obtain the plane equations of the planes corresponding to each triangle according to each normal vector and the camera coordinate of any vertex sparse point of the plane corresponding to the normal vector; perform rasterization processing on the planes corresponding to each triangle to obtain the pixel points of each plane; obtain the depth values of the pixel points of each plane according to the depth range, each plane equation, and the camera internal parameters to obtain the depth map.

[0117] In some embodiments, the coordinates of each vertex sparse point can be homogeneous coordinates. Converting the coordinates of each vertex sparse point to the camera coordinates includes: by to obtain the camera coordinates. Where, [X Y Z] is the camera coordinate of the vertex sparse point after conversion; specifically, X is the value of the x-axis in the camera coordinate system, Y is the value of the y-axis in the camera coordinate system, Z is the value of the y-axis in the camera coordinate system, X and Y jointly describe the projection position of the vertex sparse point on the image plane, that is, the offset of the vertex sparse point relative to the optical center of the camera, and Z describes the depth information of the vertex sparse point from the optical center of the camera; [u v 1] is the homogeneous coordinate of the vertex sparse point before conversion; f x is the preset first focal length; f y is the preset second focal length; c x is the offset of the camera optical axis on the x-axis of the image coordinate system; c y is the offset of the camera optical axis on the y-axis of the image coordinate system; H is the sparse depth value corresponding to the vertex sparse point.

[0118] The preset first focal length is the focal length corresponding to one image in the matching image pair. The preset second focal length is the focal length corresponding to the other image in the matching image pair.

[0119] In some embodiments, obtaining the normal vector of the plane corresponding to each triangle according to each camera coordinate includes: by calculating n m =(p 1m -p 0m )×(p 2m -p 0m ), obtaining the normal vector of the plane corresponding to the m-th triangle. Wherein, n m is the normal vector of the plane corresponding to the m-th triangle; p 0m is the first camera coordinate of the plane corresponding to the m-th triangle; p 1m is the second camera coordinate of the plane corresponding to the m-th triangle; p 2m is the third camera coordinate of the plane corresponding to the m-th triangle.

[0120] In some embodiments, obtaining the plane equation of the plane corresponding to each triangle according to each normal vector and the camera coordinate of any vertex sparse point of the plane corresponding to the normal vector respectively includes: by calculating obtaining the unit vector of the normal vector of the plane corresponding to the m-th triangle. Wherein, n cm is the unit vector of the normal vector of the plane corresponding to the m-th triangle; n m is the normal vector of the plane corresponding to the m-th triangle. By calculating obtaining the plane equation of the plane corresponding to the m-th triangle. Wherein, L m is the plane equation of the plane corresponding to the m-th triangle; p m is the camera coordinate of any vertex of the plane corresponding to the m-th triangle.

[0121] In some embodiments, obtaining the depth value of each pixel point of each plane according to the depth range, each plane equation and the camera internal parameters respectively to obtain a depth map includes: determining the region of interest according to the depth range; within the region of interest, by calculating obtaining the depth value of the u-th pixel point. Wherein, Z u is the depth value of the u-th pixel point; P u is the pixel coordinate of the u-th pixel point in the plane where it is located; K is the camera internal parameter; Pu represents the u-th pixel point.

[0122] Specifically, in the adaptive propagation module, each pixel corresponds to Ep depth hypotheses; by calculating obtaining the depth value of the u-th pixel after adaptive propagation. Wherein, D P (p u) is the depth value of the u-th pixel after adaptive propagation; D represents the depth map before adaptive propagation; pu represents the depth value of the u-th pixel; oi represents the two-dimensional fixed offset corresponding to the u-th pixel; Δoi(p u ) represents the two-dimensional offset term corresponding to the u-th pixel. Ep depth hypotheses, oi, and Δoi(p u ) are obtained by learning through the 2D CNN layer in the adaptive propagation module.

[0123] In some embodiments, the adaptive matching cost module calculates the similarity between pixel points through the following method, including: determining a reference frame and a neighborhood frame; establishing a forward parallel plane under the depth hypothesis of the reference frame; using a preset differentiable homography transformation to map the neighborhood frame into the forward parallel plane to obtain a mapped image; obtaining the image features of the mapped image and the image features of the reference frame; and obtaining the similarity between each pixel point according to the image features of the mapped image and the image features of the reference frame.

[0124] Further, determining a reference frame and a neighborhood frame includes: selecting one image as the reference frame and other images as the neighborhood frames. Or, in a pair of matching images, selecting one image as the reference frame and the other image as the neighborhood frame.

[0125] Further, using a preset differentiable homography transformation to map the neighborhood frame into the forward parallel plane includes: by calculating obtaining the coordinates of the pixel points in the image of the α-th neighborhood frame mapped into the forward parallel plane under the b-th depth hypothesis. Where P α,b is the set of the coordinates of the pixel points in the image of the α-th neighborhood frame mapped into the forward parallel plane under the b-th depth hypothesis; K α is the camera intrinsic parameter of the α-th neighborhood frame; R 0,α is the camera extrinsic parameter of the reference frame; t 0,α is the camera extrinsic parameter of the α-th neighborhood frame; d b represents the b-th depth hypothesis of the reference frame; K0 is the camera intrinsic parameter of the reference frame.

[0126] The image features of the reference frame are the first features of each pixel point in the reference frame; the image features of the mapped image are the second features of each pixel point in the mapped image.

[0127] Further, obtaining the similarity between each pixel point according to the image features of the mapped image and the image features of the reference frame includes: dividing the preset feature channels into G groups; by calculating obtaining the similarity between each pixel point. Where C is the number of feature channels; S i (p,j) gis the similarity between each pixel under the g-th group of feature channels, that is, the matching cost or group similarity vector of each pixel under the g-th group of feature channels; F0(p) g is the first feature of pixel p in the reference frame under the g-th group of feature channels; F0(p i,j ) g is the corresponding feature when pixel p is mapped to the neighborhood frame i under the g-th group of feature channels.

[0128] In some embodiments, the adaptive matching cost module is further configured to perform adaptive spatial cost aggregation on each pixel. In this way, it is possible to avoid aggregating edge pixels and adopt adaptive spatial cost aggregation.

[0129] Furthermore, 3D reconstruction is performed based on the depth map to obtain a 3D model, including: obtaining the vertex coordinates corresponding to the footprint to be extracted and the color information corresponding to the vertex coordinates based on the depth map; integrating the vertex coordinates and the color information to obtain the 3D model. In this way, the vertex coordinates corresponding to the footprint to be extracted and the color information corresponding to the vertex coordinates are obtained through the depth map, and then the vertex coordinates and the color information are integrated to obtain the 3D model, thereby realizing the 3D reconstruction of the footprint to be extracted.

[0130] Furthermore, obtaining the vertex coordinates corresponding to the footprint to be extracted and the color information corresponding to the vertex coordinates based on the depth map includes: filtering the depth map using a preset geometric consistency algorithm to obtain a plurality of geometric consistency masks and the reprojected depth map; calculating the average depth according to each geometric consistency mask and the reprojected depth map; optimizing the reprojected depth map according to the average depth; calculating the vertex coordinates corresponding to the footprint to be extracted and the color information corresponding to the vertex coordinates according to the optimized depth map.

[0131] It should be noted that the geometric consistency masks are saved as image files for subsequent recording and analysis. Optimizing the reprojected depth map according to the average depth means performing smoothing processing, filling processing, and / or calibration processing, etc. on the reprojected depth map using the average depth to optimize the reprojected depth map.

[0132] Specifically, the preset geometric consistency algorithm includes: determining a reference view and a source view in a pair of matching images; generating first pixel coordinates of all pixel points in the reference view using the meshgrid function in the numpy library; calculating first three-dimensional point coordinates corresponding to the first pixel coordinates of the pixel points in the reference view according to the camera internal parameters and depth map corresponding to the reference view using the matmul function in the numpy library; projecting the first three-dimensional point coordinates corresponding to the first pixel coordinates of the pixel points in the reference view into the source view according to the camera external parameters corresponding to the reference view and the camera external parameters corresponding to the source view using a preset matmul function to obtain second three-dimensional point coordinates after projection; obtaining second pixel coordinates corresponding to the second three-dimensional point coordinates after projection in the source view according to the camera internal parameters corresponding to the source view using a preset matmul function; obtaining a first depth map after reprojection from the depth map corresponding to the source view according to the second pixel coordinates using the remap function of OpenCV; obtaining third three-dimensional point coordinates corresponding to the source view according to the first depth map after reprojection and the camera internal parameters corresponding to the source view; projecting the third three-dimensional point coordinates corresponding to the source view back into the reference view according to the camera external parameters corresponding to the reference view and the camera external parameters corresponding to the source view to obtain fourth three-dimensional point coordinates after projection; obtaining third pixel coordinates corresponding to the fourth three-dimensional point coordinates in the reference view according to the camera internal parameters corresponding to the reference view; obtaining a second depth map after projection according to the third pixel coordinates; obtaining the Euclidean distance between the reprojection point and the original point according to the first pixel coordinates and the third pixel coordinates; wherein, the first pixel coordinates are the original points; the third pixel coordinates are the reprojection points; obtaining the absolute difference between the reprojection depth and the original depth according to the first depth map and the second depth map using the abs function in the numpy library; wherein, the depth of the first depth map is the original depth; the depth of the second depth map is the reprojection depth; determining the ratio of the absolute difference to the original depth of the depth map as the relative depth difference; in the case where the Euclidean distance between the reprojection point and the original point is less than a preset distance and the relative depth difference is less than a preset depth, generating a boolean mask using the logical_and function in the preset numpy library; determining the boolean mask as each geometric consistency mask.

[0133] Further, calculating the average depth according to each geometric consistency mask and the depth map after reprojection includes: obtaining the first sum value of the depth values of each depth map; obtaining the second sum value of the geometric consistency masks; obtaining the third sum value of the second sum value and 1; obtaining the quotient value between the first sum value and the third sum value; determining the quotient value as the average depth.

[0134] In some embodiments, the vertex coordinates corresponding to the footprint to be extracted and the color information corresponding to the vertex coordinates are calculated according to the optimized depth map using the concatenate function in the numpy library, which will not be elaborated here.

[0135] See Figure 4 , Figure 4 which is a flowchart of a method for footprint extraction shown in an exemplary embodiment of the present application.

[0136] As Figure 4 shown, in an exemplary embodiment, the method for footprint extraction at least includes steps S401 to S412, which are introduced in detail as follows:

[0137] Step S401, when the shooting angle of the camera is perpendicular to the footprint to be extracted, obtain a first image perpendicular to the direction of the footprint to be extracted.

[0138] Step S402, adjust the shooting angle of the camera to a preset angle.

[0139] Step S403, based on the shooting angle, control the camera to circularly shoot the footprint to be extracted, and obtain a plurality of second images in the direction corresponding to the preset angle.

[0140] Step S404, determine the first image and the second images as footprint images.

[0141] Step S405, perform feature extraction on each footprint image to obtain the feature information of each footprint image.

[0142] Step S406, establish a matching relationship between each footprint image to obtain a plurality of matching image pairs.

[0143] Step S407, perform feature point matching on each matching image pair according to the feature message to determine the matching points corresponding to each matching image pair.

[0144] Step S408, perform sparse reconstruction based on each matching point to obtain a reconstruction result.

[0145] Step S409, obtain reconstruction parameters according to the reconstruction result.

[0146] Step S410, input the reconstruction parameters into a preset depth map estimation model to obtain a depth map corresponding to the footprint to be extracted.

[0147] Step S411, obtain the vertex coordinates corresponding to the footprint to be extracted and the color information corresponding to the vertex coordinates based on the depth map.

[0148] Step S412, integrate the vertex coordinates and the color information to obtain a three-dimensional model.

[0149] In this way, by obtaining a first image perpendicular to the footprint to be extracted and a plurality of second images taken annularly in the direction corresponding to a preset angle, footprint images of the footprint to be extracted in a plurality of preset directions are obtained. Then, feature extraction is performed on each footprint image to obtain the feature information of each footprint image. Next, a plurality of matching image pairs are determined based on each footprint image, and sparse reconstruction is performed based on the matching points of each matching image pair to obtain a reconstruction result. A depth map corresponding to the footprint to be extracted is obtained according to the reconstruction result, and the vertex coordinates corresponding to the footprint to be extracted and the color information corresponding to the vertex coordinates are obtained based on the depth map. Then, the vertex coordinates and the color information are integrated to obtain a three-dimensional model. In this way, by adjusting the shooting angle of the camera, a plurality of footprint images in multiple directions are obtained. A large number of images with different inclination angles from the footprint plane and equidistant from the center of the footprint are collected, so that the footprint images can include the features of the footprint to be extracted from various angles, facilitating sparse reconstruction first and then stereo model reconstruction using the footprint images of the footprint to be extracted. Compared with extracting the footprint by plaster casting, taking pictures can preserve the features of the footprint to be extracted and is not easy to damage the footprint to be extracted, thus enabling accurate and rapid extraction of the footprint to be extracted.

[0150] In some embodiments, please refer to Figure 5 , Figure 5 which is a block diagram of a device for footprint extraction shown in an exemplary embodiment of the present application.

[0151] As Figure 5 shown, the exemplary device for footprint extraction includes:

[0152] An image acquisition module 501, configured to acquire footprint images of the footprint to be extracted in a plurality of preset directions;

[0153] A feature extraction module 502, configured to perform feature extraction on each footprint image to obtain the feature information of each footprint image;

[0154] A reconstruction module 503, configured to perform stereo model reconstruction according to each feature message to obtain a three-dimensional model of the footprint to be extracted.

[0155] In an exemplary embodiment, the image acquisition module 501 includes:

[0156] A first image acquisition sub-module, configured to acquire a first image in the direction perpendicular to the footprint to be extracted when the shooting angle of the camera is perpendicular to the footprint to be extracted;

[0157] An adjustment sub-module, configured to adjust the shooting angle of the camera to a preset angle;

[0158] A second image acquisition sub-module, configured to control the camera to take annular pictures of the footprint to be extracted based on the shooting angle to obtain a plurality of second images in the direction corresponding to the preset angle;

[0159] An image determination sub-module, configured to determine the first image and the second image as footprint images.

[0160] In an exemplary embodiment, the reconstruction module 503 includes:

[0161] A first sparse reconstruction sub-module, configured to perform sparse reconstruction according to the feature message to obtain a reconstruction result;

[0162] A reconstruction parameter acquisition sub-module, configured to acquire reconstruction parameters according to the reconstruction result;

[0163] An input sub-module, configured to input the reconstruction parameters into a preset depth map estimation model to obtain a depth map corresponding to the footprint to be extracted;

[0164] A three-dimensional reconstruction sub-module, configured to perform three-dimensional reconstruction based on the depth map to obtain a three-dimensional model.

[0165] In an exemplary embodiment, the first sparse reconstruction sub-module includes:

[0166] An image matching sub-module, configured to establish a matching relationship between each footprint image to obtain a plurality of matching image pairs;

[0167] A feature point matching sub-module, configured to perform feature point matching on each matching image pair according to the feature message to determine the matching points corresponding to each matching image pair;

[0168] A second sparse reconstruction sub-module, configured to perform sparse reconstruction based on each matching point to obtain a reconstruction result.

[0169] In an exemplary embodiment, it is applied to a footprint extraction device; the footprint extraction device includes a camera; the reconstruction parameter acquisition sub-module includes:

[0170] A first acquisition sub-module, configured to acquire the camera parameters of the camera and the image information of each footprint image;

[0171] A camera extrinsic parameter acquisition sub-module, configured to acquire the camera extrinsic parameters according to the image information;

[0172] A depth range acquisition sub-module, configured to acquire the depth range according to the extrinsic parameter matrix and the sparse point information;

[0173] A selection sub-module, configured to acquire the similarity scores of each matching image pair and select the matching image pair with the highest similarity based on the similarity scores among each matching image pair;

[0174] A reference sparse point information acquisition sub-module, configured to acquire the sparse point information of the selected matching image pair to obtain reference sparse point information;

[0175] A reconstruction parameter determination sub-module, configured to determine camera internal parameters, camera external parameters, depth range, and reference sparse point information as reconstruction parameters.

[0176] In an exemplary embodiment, the selection sub-module includes:

[0177] An intersection sparse point acquisition sub-module, configured to acquire intersection sparse points corresponding to each matching image pair based on sparse point information;

[0178] An included angle acquisition sub-module, configured to calculate the included angles between the corresponding intersection sparse points and the camera center corresponding to the camera for each matching image pair;

[0179] A score contribution acquisition sub-module, configured to calculate the score contributions of each intersection sparse point according to the respective included angles;

[0180] A similarity score acquisition sub-module, configured to determine the sum of the score contributions of each intersection sparse point as the similarity score of the matching image pair.

[0181] In some embodiments, the 3D reconstruction sub-module includes:

[0182] A second acquisition sub-module, configured to acquire the vertex coordinates corresponding to the footprint to be extracted and the color information corresponding to the vertex coordinates based on the depth map;

[0183] An integration sub-module, configured to integrate the vertex coordinates and the color information to obtain a 3D model.

[0184] It should be noted that the device for footprint extraction provided in the above embodiments belongs to the same concept as the method for footprint extraction provided in the above embodiments. The specific manners in which each module and unit perform operations have been described in detail in the method embodiments and will not be elaborated here. In practical applications, the device for footprint extraction provided in the above embodiments can, according to needs, allocate the above functions to different functional modules, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this will not be limited here either.

[0185] An embodiment of the present application also provides a footprint extraction device, including: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the footprint extraction device to implement the method for footprint extraction provided in each of the above embodiments.

[0186] Figure 6 The structure diagram of a computer system suitable for implementing the footprint extraction device in the embodiments of the present application is shown. It should be noted that Figure 6The computer system 600 of the footprint extraction device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0187] As Figure 6 shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the Read-Only Memory (ROM) 602 or the program loaded from the storage section 608 into the Random Access Memory (RAM) 603, such as executing the method described in the above embodiments. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0188] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including, for example, a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as required. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as required so that the computer program read from it can be installed into the storage section 608 as required.

[0189] Specifically, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 609 and / or installed from the removable medium 611. When the computer program is executed by the Central Processing Unit (CPU) 601, various functions defined in the system of the present application are executed.

[0190] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The computer program included on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Among them, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0192] The units involved in the embodiments described in this application can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation on the units themselves in certain cases.

[0193] Another aspect of this application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method for extracting footprints as described above. The computer-readable storage medium may be included in the footprint extraction device described in the above embodiments, or may exist separately without being assembled into the footprint extraction device.

[0194] Another aspect of this application also provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for extracting footprints provided in the above various embodiments.

[0195] The above content is only a preferred exemplary embodiment of this application, and is not used to limit the implementation of this application. Those of ordinary skill in the art can easily make corresponding adaptations or modifications according to the main concept and spirit of this application. Therefore, the protection scope of this application should be subject to the protection scope required by the claims.

Claims

1. A method for footprint extraction, characterized in that, Comprising: Obtaining footprint images of the footprint to be extracted in multiple preset directions; Performing feature extraction on each of the footprint images to obtain feature information of each of the footprint images; Performing stereo model reconstruction based on each of the feature messages to obtain a three-dimensional model of the footprint to be extracted.

2. The method according to claim 1, wherein Applied to a footprint extraction device; the footprint extraction device includes a camera; the obtaining footprint images of the footprint to be extracted in multiple preset directions includes: When the shooting angle of the camera is perpendicular to the footprint to be extracted, obtaining a first image perpendicular to the direction of the footprint to be extracted; Adjusting the shooting angle of the camera to a preset angle; Based on the shooting angle, controlling the camera to circularly shoot the footprint to be extracted to obtain a plurality of second images in the direction corresponding to the preset angle; Determining the first image and the second images as the footprint images.

3. The method according to claim 1, wherein The performing stereo model reconstruction based on each of the feature messages to obtain a three-dimensional model of the footprint to be extracted includes: Performing sparse reconstruction based on the feature message to obtain a reconstruction result; Obtaining reconstruction parameters according to the reconstruction result; Inputting the reconstruction parameters into a preset depth map estimation model to obtain a depth map corresponding to the footprint to be extracted; Performing three-dimensional reconstruction based on the depth map to obtain the three-dimensional model.

4. The method according to claim 3, characterized in that, The performing sparse reconstruction based on the feature message to obtain a reconstruction result includes: Establishing a matching relationship between each of the footprint images to obtain a plurality of matching image pairs; Performing feature point matching on each of the matching image pairs according to the feature message to determine matching points corresponding to each of the matching image pairs; Performing sparse reconstruction based on each of the matching points to obtain the reconstruction result.

5. The method according to claim 3, wherein Applied to a footprint extraction device; the footprint extraction device includes a camera; the reconstruction result includes sparse point information; the obtaining reconstruction parameters according to the reconstruction result includes: Obtaining camera parameters of the camera and image information of each of the footprint images; Obtaining external camera parameters according to the image information; Obtaining a depth range according to the external parameter matrix and the sparse point information; Obtaining a similarity score for each of the matching image pairs, and selecting the matching image pair with the highest similarity based on the similarity score among each of the matching image pairs; Obtaining sparse point information of the selected matching image pair to obtain reference sparse point information; Determining the camera internal parameters, the external camera parameters, the depth range, and the reference sparse point information as the reconstruction parameters.

6. The method according to claim 5, wherein The obtaining a similarity score for each of the matching image pairs includes: Obtaining intersection sparse points corresponding to each of the matching image pairs based on the sparse point information; For each matching image pair, calculating an angle between each of the corresponding intersection sparse points and the camera center corresponding to the camera; Calculating score contributions of each of the intersection sparse points according to each of the angles; Determining the sum of the score contributions of each of the intersection sparse points as the similarity score of the matching image pair.

7. The method according to claim 3, wherein The performing three-dimensional reconstruction based on the depth map to obtain the three-dimensional model includes: Obtaining vertex coordinates corresponding to the footprint to be extracted and color information corresponding to the vertex coordinates based on the depth map; Integrating the vertex coordinates and the color information to obtain the three-dimensional model.

8. A device for extracting footprints, characterized in that, Comprising: An image acquisition module, configured to acquire footprint images of a footprint to be extracted in a plurality of preset directions; A feature extraction module, configured to perform feature extraction on each of the footprint images to obtain feature information of each of the footprint images; A reconstruction module, configured to perform three-dimensional model reconstruction according to each of the feature messages to obtain a three-dimensional model of the footprint to be extracted.

9. A footprint extraction device, characterized in that, Comprising: One or more processors; A storage device for storing one or more programs, which when executed by the one or more processors, cause the footprint extraction device to implement the method for extracting a footprint according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Computer-readable instructions are stored thereon, which when executed by a processor of a computer, cause the computer to execute the method for extracting a footprint according to any one of claims 1 to 7.

Images