Map creation method and apparatus

Abstract

The application relates to a map creation method and device. The method comprises the following steps: acquiring a first shooting image and a second shooting image shot by a binocular camera for a target scene; inputting the first shooting image and the second shooting image into a pre-trained point-line detection model, acquiring first key points and first lines and second key points and second lines through the point-line detection model, and acquiring key point matching information; obtaining line matching information based on the key point matching information, line parameters of the first lines and the second lines; acquiring three-dimensional coordinates of scene lines contained in the target scene according to the line matching information, obtaining camera poses of the binocular camera based on the three-dimensional coordinates, and generating a scene map for the target scene by using the camera poses. The method can reduce the influence of light and environmental factors, thereby improving the robustness of the image matching algorithm and the accuracy of the created map.

Description

Technical Field

[0001] This application relates to the field of map building technology, and in particular to a map creation method, apparatus, visual odometry, storage medium and computer program product. Background Technology

[0002] With the development of mapping technology, a simultaneous localization and mapping (SLAM) technique has emerged. This technique uses sensors such as cameras, LiDAR, and inertial measurement units (IMUs) to collect environmental information, and then fuses this information through algorithms to determine the device's position in an unknown environment and construct an environmental map. Therefore, it can be widely used in various fields such as autonomous driving, drones, and robotics. Currently, the map creation process typically requires taking pictures of the same scene from different perspectives, then combining image matching technology and IMU pre-integration to determine the relative pose of the camera between the captured images, and finally using the relative pose to create a map.

[0003] In the above process, image matching techniques typically rely on SIFT features, or scale-invariant features, which are 128-dimensional feature vectors. During matching, the similarity between two feature vectors is primarily evaluated based on the Euclidean distance. The matching process also requires nearest neighbor search to accurately locate the key points for the best match.

[0004] However, the above image matching methods are often susceptible to lighting and environmental factors, especially when dealing with weak texture scenes, changes in lighting conditions, or texture deformation caused by perspective changes. The robustness of image matching algorithms is low, resulting in low accuracy of the created maps. Summary of the Invention

[0005] Therefore, it is necessary to provide a map creation method, apparatus, visual odometry, computer-readable storage medium, and computer program product that can improve the accuracy of the created maps, in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a map creation method, including:

[0007] Acquire the first and second images captured by the binocular camera of the target scene;

[0008] The first captured image and the second captured image are input into a pre-trained point and line detection model. The first key point and the first line contained in the first captured image and the second key point and the second line contained in the second captured image are obtained through the point and line detection model. Based on the state information of the first key point and the second key point, the key point matching information of the first captured image and the second captured image is obtained.

[0009] Based on the key point matching information, the line parameters of the first line and the second line, line matching information between the first captured image and the second captured image is obtained;

[0010] Based on the line matching information, the three-dimensional coordinates of the scene lines contained in the target scene are obtained, and based on the three-dimensional coordinates, the camera pose of the stereo camera is obtained, and a scene map for the target scene is generated using the camera pose.

[0011] Secondly, this application also provides a map creation apparatus, comprising:

[0012] The image acquisition module is used to acquire the first and second images captured by the binocular camera of the target scene;

[0013] The key point matching acquisition module is used to input the first captured image and the second captured image into a pre-trained point and line detection model, obtain the first key point and the first line contained in the first captured image and the second key point and the second line contained in the second captured image through the point and line detection model, and obtain the key point matching information of the first captured image and the second captured image based on the state information of the first key point and the second key point.

[0014] The line matching acquisition module is used to obtain line matching information between the first captured image and the second captured image based on the key point matching information, the line parameters of the first line and the second line;

[0015] The scene map construction module is used to obtain the three-dimensional coordinates of the scene lines contained in the target scene according to the line matching information, obtain the camera pose of the stereo camera based on the three-dimensional coordinates, and generate a scene map for the target scene using the camera pose.

[0016] Thirdly, this application also provides a visual odometry system, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any embodiment of the first aspect.

[0017] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any embodiment of the first aspect.

[0018] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any embodiment of the first aspect.

[0019] The aforementioned map creation method, apparatus, visual odometry, storage medium, and computer program product acquire a first and a second image of a target scene using a binocular camera via visual odometry; input the first and second images into a pre-trained point and line detection model, and obtain a first keypoint and a first line contained in the first image, and a second keypoint and a second line contained in the second image through the point and line detection model; obtain keypoint matching information between the first and second images based on the state information of the first and second keypoints; obtain line matching information between the first and second images based on the keypoint matching information and the line parameters of the first and second lines; obtain the three-dimensional coordinates of the scene lines contained in the target scene based on the line matching information, obtain the camera pose of the binocular camera based on the three-dimensional coordinates, and generate a scene map of the target scene using the camera pose. This application acquires first and second images of a target scene using a binocular camera via visual odometry. It then utilizes a pre-trained point and line detection model to detect first keypoints and first lines in the first image, second keypoints and second lines in the second image, and keypoint matching information between the first and second images. This keypoint matching information, along with the line parameters of the first and second lines, is then used to obtain line matching information between the first and second images. This line matching information is then used to obtain the 3D coordinates of the scene lines in the target scene, thereby determining the camera pose of the binocular camera to generate a scene map of the target scene. Compared to existing technologies that use SIFT features for matching, this application uses a point and line detection model to detect keypoints and lines and output keypoint matching information. This keypoint matching information is then used to obtain line matching information, thus reducing the influence of lighting and environmental factors and improving the robustness of the image matching algorithm to enhance the accuracy of the created map. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a map creation method in one embodiment;

[0022] Figure 2 This is a schematic diagram of the process for obtaining line matching information in one embodiment;

[0023] Figure 3This is a schematic diagram of the process for obtaining point-line matching information in one embodiment;

[0024] Figure 4 This is a schematic diagram of the process of obtaining line matching based on point-line and key point matching in one embodiment;

[0025] Figure 5 This is a schematic diagram illustrating the process of acquiring captured images in one embodiment;

[0026] Figure 6 This is a schematic diagram of the spatiotemporal alignment process for an initial captured image in one embodiment;

[0027] Figure 7 This is a structural block diagram of a map creation device in one embodiment;

[0028] Figure 8 This is a diagram of the internal structure of a visual odometer in one embodiment. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] In one embodiment, such as Figure 1 As shown, a map creation method is provided. This embodiment illustrates the application of this method to visual odometry in robots or automobiles. In this embodiment, the method includes the following steps:

[0031] Step S101: Acquire the first and second images captured by the binocular camera of the target scene.

[0032] The target scene refers to the scene for which a map needs to be created. This map can be created using images captured by a binocular camera of the scene. The first and second images are images captured by the binocular camera from different perspectives of the target scene. Specifically, after the binocular camera captures the first and second images of the target scene, they can be transmitted to a visual odometry system, allowing the system to acquire both images.

[0033] Step S102: Input the first captured image and the second captured image into the pre-trained point and line detection model, obtain the first key point and the first line contained in the first captured image, and the second key point and the second line contained in the second captured image through the point and line detection model, and obtain the key point matching information of the first captured image and the second captured image based on the state information of the first key point and the second key point.

[0034] A point and line detection model refers to a pre-trained detection model used to detect key points and lines in an image. This detection model can be trained using deep learning and can also output matching information between key points in two images. This matching information indicates that the key points in the two images are the same key points. For example, if key point A in the first image and key point B in the second image have a matching relationship, it means that key point A and key point B are the same scene points in the target scene.

[0035] Specifically, the visual odometry system can be deployed with a pre-trained point and line detection model. After obtaining a first image and a second image, these images can be input into the pre-trained point and line detection model. The model then extracts key points and lines from the first image as first key points and first lines, and similarly extracts key points and lines from the second image as second key points and second lines. Furthermore, the model can extract the state information of the first and second key points. This state information can be used to characterize the state features of the first and second key points, thereby enabling the matching of each first and second key point to obtain key point matching information between the first and second images.

[0036] Step S103: Based on the key point matching information, the line parameters of the first line and the second line, obtain the line matching information between the first captured image and the second captured image.

[0037] Line matching information is used to characterize the matching information between lines in two images. This matching information indicates that the lines in the two images are the same lines, while the line parameters refer to the parameters of the first and second lines, such as the coordinate representation parameters of the lines. Specifically, after the point and line detection model outputs keypoint matching information, as well as the first and second lines, the visual odometry can also use the keypoint matching information and the line parameters of the first and second lines to obtain the line matching information between the first and second captured images.

[0038] Step S104: Based on the line matching information, obtain the three-dimensional coordinates of the scene lines contained in the target scene, and based on the three-dimensional coordinates, obtain the camera pose of the stereo camera, and use the camera pose to generate a scene map for the target scene.

[0039] Camera pose refers to the position and orientation information of the binocular camera, while scene lines are the lines contained in the target scene. These lines can be determined based on line matching information. For example, if the line matching information indicates that line A in the first captured image and line B in the second captured image are the same line, then line A and line B can represent the same scene line 1. Three-dimensional coordinates refer to the three-dimensional spatial coordinates corresponding to each scene line.

[0040] Specifically, after obtaining line matching information, the visual odometry system can identify scene lines within the target scene based on this information and further obtain the 3D coordinates of each scene line. These 3D coordinates can then be used to calculate the camera pose of the stereo camera, thereby generating a scene map of the target scene. For example, the camera pose output from the above process, along with the IMU pre-integration results, can be used to estimate the initial pose to generate an initial scene map. This initial scene map can then be optimized offline, and the optimized map can be re-localized to achieve map reconstruction. This method ultimately generates the scene map of the target scene.

[0041] In the above map creation method, a first and a second image of the target scene are acquired by a binocular camera using visual odometry. The first and second images are then input into a pre-trained point and line detection model. The model acquires the first keypoint and the first line contained in the first image, as well as the second keypoint and the second line contained in the second image. Based on the state information of the first and second keypoints, keypoint matching information between the first and second images is obtained. Based on the keypoint matching information and the line parameters of the first and second lines, line matching information between the first and second images is obtained. Based on the line matching information, the three-dimensional coordinates of the scene lines contained in the target scene are acquired. Based on the three-dimensional coordinates, the camera pose of the binocular camera is obtained. Finally, using the camera pose, a scene map of the target scene is generated. This application acquires first and second images of a target scene captured by a binocular camera using visual odometry. A pre-trained point and line detection model is then used to detect first keypoints and first lines in the first image, second keypoints and second lines in the second image, and keypoint matching information between the first and second images. This keypoint matching information, along with the line parameters of the first and second lines, is then used to obtain line matching information between the first and second images. This line matching information is then used to obtain the 3D coordinates of the scene lines in the target scene, thereby determining the camera pose of the binocular camera to generate a scene map of the target scene. Compared to existing technologies that use SIFT features for matching, this application uses a point and line detection model to detect keypoints and lines and output keypoint matching information. This keypoint matching information is then used to obtain line matching information, thus reducing the influence of lighting and environmental factors and improving the robustness of the image matching algorithm to increase the accuracy of the created map.

[0042] In one embodiment, such as Figure 2 As shown, step S103 may further include:

[0043] Step S201: Obtain the coordinate information of each first key point and second key point, as well as the line endpoint information of each first line and second line.

[0044] The coordinate information refers to the position coordinates of the first key point and the second key point in the first and second captured images, respectively, while the line endpoint information refers to the line endpoint coordinates of the first line and the second line. In this embodiment, after the visual odometry extracts the first key point, the second key point, the first line and the second line through the point and line detection model, it can also obtain the coordinate information of each first key point and the second key point, as well as the line endpoint information of each first line and the second line.

[0045] Step S202: Obtain point-line matching information based on coordinate information, line endpoint information, and line parameters; the point-line matching information is used to characterize the matching relationship between the first key point and the first line, and the matching relationship between the second key point and the second line.

[0046] Point-line matching information is used to characterize the matching relationship between key points and lines. This matching relationship can include the matching relationship between the first key point and the first line, and the matching relationship between the second key point and the second line. Specifically, visual odometry can use the coordinate information of the first and second key points, the line endpoint information of the first and second lines, and the line parameters of the first and second lines to identify the matching relationship between key points and lines as point-line matching information.

[0047] Step S203: Obtain line matching information based on point-line matching information and key point matching information.

[0048] Finally, the visual odometry can combine the key point matching information and point-line matching information of the first and second captured images to obtain the line matching information of the first and second captured images.

[0049] In this embodiment, the coordinates of each key point detected by the point-line detection model and the endpoint information of each line can also be used to obtain point-line matching information. Then, the point-line matching information and the key point matching information obtained by the point-line detection model can be combined to obtain line matching information. Compared with obtaining line matching information through a model, the method provided in this embodiment can improve the efficiency and effectiveness of line matching.

[0050] Furthermore, the coordinate information includes both horizontal and vertical coordinates, and the line endpoint information includes the endpoint information of the first and second endpoints, with the endpoint information including both horizontal and vertical coordinates; for example... Figure 3 As shown, step S202 may further include:

[0051] Step S301: Based on the coordinate information of the current key point and the line parameters of the current line, obtain the distance between the current key point and the current line; the current key point is either the first key point or the second key point, and the current line is either the first line or the second line.

[0052] In this embodiment, the coordinate information of both the first key point and the second key point can include two parts, namely the horizontal coordinate and the vertical coordinate of the key point. For example, for a certain key point... The coordinate information can be obtained through To represent, where, Represents the x-axis, while This represents the ordinate. Similarly, the endpoint information of the first and second lines can also include two parts, namely the endpoints of the lines, i.e., the endpoint information of the first and second endpoints. For example, for a certain line... Endpoint information can be represented as The x-coordinate of the first endpoint is The vertical axis is The x-coordinate of the second endpoint is The vertical axis is .

[0053] and The line parameters can be expressed as Specifically, visual odometry can use any first or second key point as the current key point and any first or second line as the current line, and then calculate the distance between the current key point and the current line based on the coordinate information of the current key point and the line parameters of the current line.

[0054] For example, the current key point arrive distance for:

[0055]

[0056] Among them, the current key points The coordinate information is Current line The line parameters are represented as .

[0057] Step S302: If the distance is less than a preset distance threshold, obtain the endpoint information of the current first endpoint and the current second endpoint; the current first endpoint is the first endpoint of the current line, and the current second endpoint is the second endpoint of the current line.

[0058] Step S303: If the horizontal coordinate information of the current key point is located between the horizontal coordinate information of the current first endpoint and the current second endpoint, or if the vertical coordinate information of the current key point is located between the vertical coordinate information of the current first endpoint and the current second endpoint, generate point-line matching information between the current key point and the current line.

[0059] The preset distance threshold is a pre-set distance threshold. The current first endpoint refers to the first endpoint of the current line, and the current second endpoint refers to the second endpoint of the current line. If the distance between the current key point and the current line is less than the preset distance threshold, it can be determined whether the horizontal coordinate information of the current key point is between the horizontal coordinate information of the endpoints of the current first endpoint and the current second endpoint, and whether the vertical coordinate information of the current key point is between the vertical coordinate information of the endpoints of the current first endpoint and the current second endpoint. If one of the conditions is met, it can be determined that the projection of the current key point on the coordinate axis is within the projection of the endpoint of the line segment. Therefore, it can be determined that there is a point-line matching relationship between the current key point and the current line.

[0060] For example, if the preset distance threshold is 3, if the condition is met... And it satisfies the condition that the projection of the current key point on the coordinate axis lies within the projection of the endpoint of the line segment, i.e., satisfies... ,or In this case, it explains something. and lines It matches.

[0061] In this embodiment, the distance between the current key point and the current line can also be calculated. If the distance is less than a preset distance threshold, it can be further determined whether the horizontal coordinate information of the current key point is located between the horizontal coordinate information of the current first endpoint and the current second endpoint, or whether the vertical coordinate information of the current key point is located between the vertical coordinate information of the current first endpoint and the current second endpoint. If so, point-line matching information between the current key point and the current line can be generated. This method can improve the efficiency of obtaining point-line matching information.

[0062] In addition, such as Figure 4 As shown, step S203 may further include:

[0063] Step S401: Obtain the first current line and the second current line; the first current line is any one of the first lines, and the second current line is any one of the second lines;

[0064] Step S402: Based on the dot-line matching information, obtain the first key point that matches the first current line and the second key point that matches the second current line.

[0065] The first current line can refer to any one of the first lines contained in the first captured image, while the second current line refers to any one of the second lines contained in the second captured image. Specifically, after obtaining the point-line matching information, the visual odometry can also determine the first key point matching each first current line and the second key point matching each second current line based on the point-line matching information.

[0066] Step S403: Based on the key point matching information, obtain the number of matching key points between the first key point matched by the first current line and the second key point matched by the second current line;

[0067] Step S404: If the number of matching key points meets the preset conditions, generate line matching information between the first current line and the second current line.

[0068] Then, the key point matching information can be used to determine the number of key points that have a matching relationship with the first key point of the first current line and the second key point of the second current line, i.e. the number of matching key points. Then, it can be determined whether the number of matching key points meets the preset conditions. If so, the line matching information between the first current line and the second current line is generated.

[0069] For example, the point-line matching information indicates that line 1 is matched with key points A, B, and C, and line 2 is matched with key points D, E, and F. The key point matching information indicates that key points A and D are matched, key points B and E are matched, and key points C and F are matched. Therefore, the number of matched key points is 3. In this way, line segments on two images can be matched based on the point matching results of two images.

[0070] In this embodiment, the number of matching key points between any two first lines and second lines can be determined based on the point-line matching information and key point matching information. Thus, line matching information between the first lines and the second lines can be generated based on the number of matching key points. This method can improve the accuracy of line matching.

[0071] Furthermore, step S404 may further include: obtaining the confidence level between the first current line and the second current line based on the number of matching key points; and generating line matching information between the first current line and the second current line when the number of matching key points is greater than a preset number threshold and the confidence level is greater than a preset confidence threshold.

[0072] The confidence level between the first current line and the second current line is used to characterize the confidence that the first current line and the second current line are the same line. In this embodiment, the number of matching key points satisfying the preset conditions may include the following two: the number of matching key points is greater than a preset number threshold, and the confidence level between the first current line and the second current line obtained based on the number of matching key points is greater than a preset confidence threshold. Only when both of the above conditions are satisfied at the same time will the line matching information between the first current line and the second current line be generated.

[0073] For example, the first current line With the second current line confidence levels between It can be calculated using the following formula:

[0074]

[0075] in, yes Point features and The number of point features matched, i.e., the number of key points matched. , They are and The point characteristic number, if and ,in and There are two thresholds; if they are met, then... and They can be considered as the same straight line, therefore they can be matched.

[0076] In this embodiment, the confidence level between the first current line and the second current line can also be obtained by matching the number of key points, thereby determining whether the number of matching key points is greater than a preset number threshold and whether the confidence level is greater than a preset confidence threshold, so as to obtain line matching information. This method can further improve the accuracy of line matching.

[0077] In one embodiment, such as Figure 5 As shown, step S101 may further include:

[0078] Step S501: Obtain the first initial captured image and the second initial captured image of the target scene captured by the binocular camera, and obtain the event camera data stream captured by the event camera of the target scene;

[0079] Step S502: Use the event camera data stream to perform spatiotemporal alignment processing on the first initial captured image and the second initial captured image to obtain the aligned first initial captured image and the second initial captured image.

[0080] Here, the first initial captured image and the second initial captured image refer to the captured images directly captured by the binocular camera on the target scene. In this embodiment, the first captured image and the second captured image used to input the point and line detection model can be obtained by preprocessing the first initial captured image and the second initial captured image captured by the binocular camera. The event camera data stream refers to the event data stream captured by the event camera on the target scene. This event data stream can be used to realize the spatiotemporal alignment processing of the captured initial images. The aligned first initial captured image and the second initial captured image refer to the first initial captured image and the second initial captured image after the spatiotemporal alignment processing is completed.

[0081] Specifically, in addition to acquiring the first and second initial images directly captured by the binocular camera of the target scene, the visual odometry system can also acquire event data streams captured by the event camera of the target scene, as the event camera data stream. The event camera data stream can then be used to perform spatiotemporal alignment processing on the first and second initial images, resulting in the spatiotemporally aligned first and second initial images.

[0082] Step S503: Input the aligned first initial captured image and the second initial captured image into the pre-trained dynamic region segmentation model, and remove the dynamic region from the aligned first initial captured image and the second initial captured image to obtain the first captured image and the second captured image.

[0083] The dynamic region segmentation model refers to a neural network model used to remove dynamic object regions contained in captured images. This model can use dynamic semantic segmentation algorithms to output the probability of image pixels belonging to dynamic objects in the aligned first initial captured image and the second initial captured image, thereby realizing the identification of dynamic object regions. It can also remove dynamic regions in the aligned first initial captured image and the second initial captured image to obtain the first captured image and the second captured image.

[0084] In this embodiment, after the visual odometry acquires the first initial image and the second initial image captured by the binocular camera for the target scene, it can also acquire the event camera data stream captured by the event camera for the target scene. The event camera data stream is then used to align the first initial image and the second initial image, and the aligned image is input into a dynamic region segmentation model to remove dynamic regions, so as to obtain the first and second images used to realize point and line detection. This method can improve the robustness of map creation in dynamic environments and further improve the accuracy of scene map creation.

[0085] Furthermore, the number of first and second initial images captured by the binocular cameras for the target scene is multiple, each corresponding to a different camera frame; such as Figure 6 As shown, step S502 may further include:

[0086] Step S601: Obtain the current first initial captured image and the current second initial captured image corresponding to the current camera captured frame.

[0087] In this embodiment, the first initial image and the second initial image captured by the binocular camera for the target scene can be multiple, and each initial image corresponds to a different camera frame. The current camera frame can refer to any one of the multiple camera frames, and the current first initial image and the current second initial image refer to the first initial image and the second initial image corresponding to the current camera frame.

[0088] Specifically, the visual odometry can determine any one of multiple camera frames as the current camera frame, and then use the first initial image and the second initial image corresponding to the current camera frame as the current first initial image and the current second initial image, respectively.

[0089] Step S602: Based on the timestamp corresponding to the current camera frame, obtain the current event data matching the timestamp from the event camera data stream, and the first event coordinates corresponding to the current event data in the first coordinate system; the first coordinate system is the coordinate system of the event camera.

[0090] Current event data refers to the event data in the event camera data stream that matches the current camera frame. This matching can be achieved using timestamps, which can be used to represent the shooting time. The first coordinate system refers to the coordinate system of the event camera, and the first event coordinates refer to the position coordinates of the current event data in the first coordinate system.

[0091] Specifically, the event camera data stream can contain the location coordinates and timestamps of the event data. After the visual odometry obtains the timestamp corresponding to each current camera frame, it can use the timestamp to match the event data with the same timestamp from the event camera data stream as the current event data, thereby achieving time alignment, and using the location coordinates corresponding to the current event data as the first location coordinates.

[0092] Step S603: Convert the first event coordinates into the second event coordinates corresponding to the current event data in the second coordinate system, and superimpose the second event coordinates onto the current first initial captured image and the current second initial captured image to obtain the aligned first initial captured image and second initial captured image corresponding to the current camera captured frame; the second coordinate system is the coordinate system of the stereo camera.

[0093] The second coordinate system refers to the coordinate system of the stereo camera, and the second event coordinates are the position coordinates of the current event data corresponding to the second coordinate system. After obtaining the first event coordinates, the visual odometry can further transform the first event coordinates to obtain the second event coordinates corresponding to the second coordinate system. Then, the visual odometry can further superimpose the second event coordinates onto the current first initial image and the current second initial image to achieve spatial alignment, resulting in aligned first and second initial images.

[0094] For example, visual odometry can first receive a binocular RGB image and an asynchronous event stream from an event camera as input. The binocular image provides high-resolution texture information at a fixed frame rate (e.g., 30Hz), while the event camera captures brightness change events with microsecond-level precision, forming a sparse spatiotemporal signal stream. To achieve collaborative processing of multimodal data, spatiotemporal alignment is achieved through timestamp interpolation and spatial calibration. At the temporal level, a strict temporal correspondence is established between the event stream and RGB frames through linear interpolation; at the spatial level, this is based on the calibration parameter matrix. Mapping event coordinates to the RGB pixel plane eliminates viewpoint differences between sensors. This alignment process can be formally represented as:

[0095]

[0096] in and These are the intrinsic parameter matrices for the stereo camera and the event camera, respectively, ensuring spatial consistency of cross-modal features.

[0097] In this embodiment, time alignment of the asynchronous event streams of the stereo image and the event camera can also be achieved through timestamps, and coordinate transformation can be achieved through calibration parameter matrices to achieve spatial alignment of the asynchronous event streams of the stereo image and the event camera. This method can improve the accuracy of spatiotemporal alignment.

[0098] In one embodiment, step S102 may further include: inputting the first captured image and the second captured image into the point and line detection model, obtaining the first key point and the second key point through the key point detection branch of the point and line detection model; and obtaining the first line and the second line through the line detection branch of the point and line detection model.

[0099] The keypoint detection branch is a model branch in the point and line detection model used to detect keypoints in an image, while the line detection branch is a model branch in the point and line detection model used to detect lines in an image. In this embodiment, the point and line detection model may include a keypoint detection branch and a line detection branch. The keypoint detection branch is used to output the keypoints contained in the image, while the line detection branch is used to output the lines contained in the image.

[0100] Specifically, the visual odometry can input the first captured image and the second captured image into the point and line detection model, so that the key point detection branch can output the key points contained in the first captured image and the second captured image respectively, that is, output the first key point and the second key point, and the line detection branch can output the lines contained in the first captured image and the second captured image respectively, that is, output the first line and the second line.

[0101] In this embodiment, a point and line detection model containing two detection branches, namely a key point detection branch and a line detection branch, can be set up. The first key point and the second key point are output through the key point detection branch, and the first line and the second line are output through the line detection branch. This embodiment can output image key points and lines simultaneously, thereby improving the robustness of key point and line detection.

[0102] Furthermore, obtaining the first key point and the second key point through the key point detection branch of the point-line detection model can further include: obtaining each image feature point contained in the first and second captured images, as well as the state information of each image feature point, through the key point detection branch; obtaining the self-attention features and bidirectional attention features of each image feature point based on the state information of each image feature point; and obtaining the first key point and the second key point from the image feature points based on the self-attention features and bidirectional attention features of each image feature point.

[0103] Image feature points refer to the pixel feature points contained in the first and second captured images. In this embodiment, the key point detection branch extracts key points from the image by first extracting the image feature points and their state information. This state information ensures the feature information of the image feature points. Then, self-attention and cross-attention mechanisms are applied to each image feature point to obtain its self-attention and bidirectional attention features. Finally, the self-attention and bidirectional attention features of each image feature point can be used to determine the first and second key points from the image feature points.

[0104] In this embodiment, the key point detection branch can extract key points by first extracting the image feature points contained in the first and second images, as well as the state information of each image feature point, and then using the attention mechanism to obtain the self-attention features and bidirectional attention features of each image feature point, thereby achieving key point filtering. This method can improve the accuracy of key point extraction.

[0105] In one embodiment, step S104 may further include: determining the scene lines contained in the target scene and the Plück coordinates of each scene line based on the line matching information, the first line and the second line; using the Plück coordinates as the three-dimensional coordinates of each scene line, and obtaining the camera coordinates of the stereo camera based on the three-dimensional coordinates of each scene line; and obtaining the camera pose of the stereo camera based on the camera coordinates of the stereo camera.

[0106] Plücker coordinates are a coordinate system used to represent directed lines in three-dimensional space. This coordinate system uses a pair of vectors, namely a direction vector and a moment vector, to represent a straight line. Since line matching information can characterize whether the first line in the first captured image and the second line in the second captured image are the same lines, visual odometry can determine the scene lines contained in the target scene based on the line matching information and the first line contained in the first captured image and the second line contained in the second captured image, and further obtain the Plücker coordinates corresponding to each scene line.

[0107] After obtaining the Plück coordinates of each scene line, the Plück coordinates can be used as three-dimensional coordinates. The three-dimensional coordinates of each scene line can be used to calculate the camera position coordinates of the stereo camera. Thus, the camera pose of the stereo camera can be obtained using the position coordinates of the stereo camera.

[0108] In this embodiment, the three-dimensional coordinates of scene lines can also be represented by Plück coordinates, and the camera coordinates of the stereo camera can be obtained using the three-dimensional coordinates of the scene lines, so as to obtain the camera pose of the stereo camera. In this way, scene lines can be effectively represented by Plück coordinates, thereby improving the computational efficiency of scene line coordinate representation.

[0109] In one embodiment, the number of first and second images captured by the binocular camera for the target scene is multiple, each corresponding to a different camera capture frame; step S104 may further include: obtaining the target capture frame from the multiple camera capture frames, and obtaining the camera coordinates corresponding to each target capture frame; and obtaining the camera pose of the binocular camera based on each camera coordinate.

[0110] In this embodiment, the number of first and second images captured by the binocular camera of the target scene can be multiple, each corresponding to a different camera capture frame. For example, in capture frame 1, the binocular camera can capture first image 1 and second image 1; in capture frame 2, the binocular camera can capture first image 2 and second image 2, and so on. The target capture frame refers to the camera capture frame used to calculate the camera pose.

[0111] Since the camera captures multiple frames, there are multiple sets of first and second images corresponding to different camera capture frames. In this case, the first and second lines of each set of first and second images can be extracted using a point-line detection model, and the line matching information of each set of first and second images can be calculated. Then, a target frame can be selected from the multiple camera capture frames. The line matching information corresponding to the target frame is then used to obtain the 3D coordinates of the scene lines corresponding to each target frame, allowing the calculation of the camera pose for each target frame. Furthermore, the camera poses of each target frame are used to calculate the camera pose of the stereo camera. Compared to requiring the camera poses of all camera capture frames to calculate the camera pose of the stereo camera, this embodiment reduces the number of capture frames used for calculating the camera pose, thereby improving the efficiency of camera pose acquisition.

[0112] In this embodiment, the camera pose of the stereo camera can be calculated by extracting the target shooting frames and using only the camera pose corresponding to each target shooting frame. Compared with the requirement to use the camera poses of all camera shooting frames to realize the camera pose of the stereo camera, this embodiment can reduce the number of shooting frames used to calculate the camera pose, thereby improving the efficiency of camera pose acquisition.

[0113] Furthermore, the target frame is obtained from multiple camera capture frames, including: obtaining the tracking features corresponding to the current capture frame; the current capture frame is any one of the multiple camera capture frames; the tracking features are key point matching information and line matching information corresponding to the current capture frame; when the number of tracking features reaches a preset feature number threshold, and the degree of viewpoint difference between the first capture image and the second capture image corresponding to the current capture frame does not reach a preset difference degree threshold, the current capture frame is taken as the target capture frame.

[0114] In this context, tracking features refer to the keypoint matching information and line matching information corresponding to a certain camera frame. In this embodiment, the selection of target frames can be achieved as follows: First, each camera frame is taken as the current frame, and the number of tracking features in the current frame, i.e., the number of keypoint matching information and line matching information corresponding to the current frame, is determined to be whether it reaches a preset threshold. Simultaneously, the degree of viewpoint difference between the first and second images corresponding to the current frame can be determined to be whether it exceeds a preset threshold. If the number of tracking features reaches the preset threshold, i.e., the number of tracking features reaches a certain level, and the degree of viewpoint difference between the first and second images does not reach the preset threshold, i.e., the viewpoint difference between the first and second images is not significant, then the current frame is taken as the target frame. In other words, visual odometry can eliminate frames with a small number of tracking features and frames with a large viewpoint difference between the first and second images, thus retaining the remaining frames as target frames.

[0115] In this embodiment, the visual odometry can eliminate frames with a small number of tracking features and large differences in viewing angle between the first and second captured images, thereby retaining the captured frames as target captured frames. This method can improve the efficiency of target captured frame selection.

[0116] In one embodiment, a method for purely visual map initialization based on deep learning feature matching is also provided. This method applies deep learning technology to feature detection and matching, combined with traditional backend optimization techniques. Specifically, a unified neural network architecture is proposed that can simultaneously extract keypoint and line segment features. Through coupling, these features are associated, matched, triangulated, and optimized. Furthermore, this scheme integrates IMU pre-integration with the matching localization method, making the localization more stable and reliable. In this process, keypoints, line segments, and structure maps are used to effectively match query frames with the map.

[0117] The pure vision-based localization and mapping system proposed in this embodiment comprises three main modules: online binocular visual odometry (VO), offline map optimization, and online relocalization. The VO module constructs an initial map by processing a sequence of map images. Then, the initial map is optimized offline to generate an optimized map. This optimized map can be used for one-time relocalization. The first part, map construction, is particularly important for real-time performance and accuracy, and includes point and line detection, feature matching, 3D feature processing, keyframe selection, IMU pre-integration, and map initialization, specifically comprising the following steps:

[0118] 1. Dot and line detection:

[0119] In pure visual localization and mapping, a camera captures images of the same scene from different perspectives. Image matching technology is used to determine the camera's relative pose between the two captures, and the 3D structure of the scene is reconstructed accordingly. Image matching is a crucial step in this process, mainly involving two core steps: feature extraction and matching. In traditional techniques, image matching often relies on SIFT features, a 128-dimensional feature vector. Matching is primarily based on evaluating the similarity between two feature vectors using the Euclidean distance. The matching process also requires nearest neighbor (NN) search to accurately locate the best-matching feature point.

[0120] Compared to traditional methods, deep learning-driven image matching schemes demonstrate significant advantages, especially in handling weakly textured scenes, dealing with changes in lighting conditions, or texture deformation caused by viewpoint shifts, where deep learning schemes offer higher robustness. This embodiment uses the SuperPoint point detection model as a benchmark and extends its architecture with a line detection branch, while also introducing a point-to-line matching mechanism between two frames of images.

[0121] This embodiment leverages the attention mechanism in the Transformer architecture to simulate the human thought process when performing feature matching tasks, aiming to solve the feature matching problem more accurately. The model predicts partial matching relationships between sets of local features extracted from images A and B. The model is constructed from L identical layers that work together to process two sets of feature data. Each layer contains self-attention and cross-attention modules to update the representation of each feature point. Subsequently, each layer is equipped with a classifier to determine whether to terminate the inference process, thereby reducing unnecessary computational overhead. Finally, a lightweight output layer computes the partial matching result from these updated representations. In the self-attention mechanism, each feature point considers information from all other points within the same image. For each point i, the current state... Decomposed into key vectors by linear transformation and query vector Rotational encoding R is used to define the attention score between points i and j. To capture their relative positions:

[0122]

[0123] By partitioning the space into d / 2 2D subspaces and rotating and projecting them onto learnable basis vectors... This achieves positional encoding. Rotation encoding enables the model to retrieve a point j with the learned relative position. This encoding is identical across all layers and is computed only once and cached.

[0124] In cross-attention, points in each image pay attention to all points in the other image. The model computes a key for each element. However, there is no query. The bidirectional attention score is:

[0125]

[0126] To improve computational efficiency, this embodiment proposes a network hierarchy with adaptive depth and width, capable of predicting the allocation of updated states at any given level. A confidence classifier assists in determining whether to terminate the inference process. If only a few points exhibit high confidence, the inference continues to the next layer, while those points with low confidence and difficult to match are discarded. The backbone network of this detection model performs contextual enhancement processing on the input visual descriptors. When image pairs are easy to match, the predictions from earlier levels are highly consistent with those from later levels and have high confidence; at this point, these predictions can be directly output and the inference terminated. At the end of each level, the model uses a simplified multilayer perceptron (MLP) to evaluate the confidence of the predicted allocation for each point:

[0127]

[0128] For a specific level L, if the confidence level at a certain point... Exceeding the threshold If the proportion of reliable points reaches or exceeds a preset value α, the inference process is considered reliable. Threshold The accuracy of each classifier on the validation set is progressively lowered, while the α parameter is used to balance accuracy and inference time. If the termination condition is not met, points that are judged as both reliable and unmatchable contribute negligibly to the matching of other points in subsequent levels. These points are mostly distributed in obviously invisible areas of the image. Therefore, these points are filtered out in each level, and only the remaining points are passed to the next level, thereby significantly reducing the computational burden.

[0129] 2. Feature matching:

[0130] To address the issues of effectiveness and efficiency, this embodiment designs a fast and robust line matching method under illumination conditions. First, keypoints are associated with line segments based on distance. Assume that M keypoints and N line segments are detected in the image, where each keypoint is represented as... Each line segment is represented as , It is a line segment The parameters, It is an endpoint, calculate first. arrive distance for:

[0131]

[0132] if ,and The projection onto the coordinate axes lies within the projection of the endpoints of the line segment, for example... or This indicates that the points and lines are matched. Then, we find the corresponding points and lines and match them. Based on the point matching results of the two images, we match the line segments on the two images. For image k... and the image on k+1 Calculate a fraction This represents the confidence level that they are on the same line.

[0133]

[0134] in, yes Point features and The number of matching point features, , They are and The point characteristic number, if and ,in and There are two thresholds, and They can be considered as the same straight line, therefore they can match.

[0135] This coupled feature matching method makes line matching both robust to keypoint matching and does not require another line matching network, thus achieving high efficiency.

[0136] 3. 3D Feature Processing:

[0137] 3.1 3D line representation:

[0138] Using Plucker coordinates to represent 3D spatial lines, the Plucker coordinate system is mainly a relatively simple way to uniquely determine and represent straight lines in space using matrices. It can express a straight line in space using six variables: the vector difference between two points on the line and the cross product of the vectors between the two points.

[0139]

[0140] Where v is the direction vector of the line, and n is the normal vector of the plane defined by the line and the origin. Plucker coordinates are used for 3D line triangulation, transformation, and projection. It is overparameterized because it is a 6-dimensional vector, but a 3D line only has 4 degrees of freedom. During graph optimization, these extra degrees of freedom increase computational cost and lead to numerical instability in the system. Therefore, orthogonal representation is used to represent 3D lines.

[0141]

[0142] The relationship between plucker coordinates and orthogonal representation is similar to that of SO (3), where orthogonal representation can be obtained from plucker coordinates:

[0143]

[0144] in , It is a diagonal matrix, and its two proportionally defined non-zero terms can be represented by a so(2) matrix:

[0145]

[0146] 3.2 Triangulation:

[0147] Triangulation is used to determine the initial position of a three-dimensional line based on the features of two or more two-dimensional lines. In this embodiment, two line segments l1 and l2 are selected from two images, representing two observations of the same three-dimensional line. These two images may originate from a stereo camera pair at the same keyframe or from two different keyframes. By backprojecting l1 and l2, we can construct two three-dimensional planes Π1 and Π2, and the three-dimensional line is the intersection of these two planes.

[0148] However, triangulation of a 3D line is far more difficult than triangulation of a 3D point because it is more susceptible to degenerate motion. Therefore, if the aforementioned method fails, this embodiment also provides a second line triangulation method, which uses points to deduce the 3D line. To initialize a 3D line, the two triangulated points X1 and X2 that belong to the line and are closest to it on the image plane are selected, thereby enabling the calculation of the Plucker coordinates of the line:

[0149]

[0150] 3.3 Reprojection:

[0151] Reprojection is used to calculate reprojection error. Plucker coordinates are used to transform and reproject 3D lines. First, the 3D lines are transformed from the world frame to the camera frame:

[0152]

[0153] In the formula, and These are the Plucker coordinates of the 3D line in the camera frame and the world frame, respectively. For the rotation matrix from the world frame to the camera frame, Translation vector, Let the transformation matrix of the 3D line be from the world frame to the camera frame, and then the 3D line... Through the linear projection matrix Projected onto the image plane:

[0154]

[0155] in It is a reprojected 2D line on the image plane. represent The first three lines.

[0156] 4. Keyframe selection:

[0157] The learning-based data association method used in this embodiment can track two frames with a large baseline. Therefore, unlike the frame-by-frame tracking strategy used in other VO or SLAM systems, it only matches the current frame with the last key frame. This strategy can reduce the accumulated tracking error.

[0158] Therefore, the selection of keyframes is crucial in this embodiment. On the one hand, it is desirable to make keyframes sparse to reduce the consumption of computational resources. On the other hand, the sparser the keyframes, the greater the possibility of tracking failure. To balance efficiency and tracking robustness, frames with fewer tracking features and larger viewpoint differences between captured images can be discarded, and the remaining frames can be used as keyframes.

[0159] 5. IMU pre-integration:

[0160] Pre-integrate the IMU measurements between frame i and frame j:

[0161]

[0162]

[0163]

[0164] in and These are angular acceleration and acceleration, respectively. and It is sensor bias, through and It is modeled as a constant between two frames. and If the noise is Gaussian, then the IMU residual is defined as:

[0165]

[0166]

[0167]

[0168]

[0169] Here, g represents the gravity vector in the world coordinate system, and an optimizer is used to minimize the cost function. If the residuals corresponding to outliers of points and lines are too large, the optimization will be rejected.

[0170] 6. Initial Map:

[0171] In this embodiment, the map building process employs an offline optimization method. This means that keyframes, map points, and 3D lines are temporarily stored on disk, awaiting further optimization after the visual odometry process is completed. Specifically, for each keyframe, its index, pose, keypoints, keypoint descriptors, line features, and connectors are saved. Simultaneously, the mapping relationship between 2D and 3D features is recorded. To improve the efficiency of map data storage, loading, and transmission across different devices, all this information is stored in binary format. This not only speeds up processing but also significantly reduces the initial map data size, making it much smaller than the original data.

[0172] In this embodiment, by applying deep learning technology to point and line detection, combined with backend optimization of traditional methods, the efficiency of feature extraction, matching, and optimization is effectively improved. Simultaneously, deep learning enhances the robustness of feature matching, enabling more stable localization, especially under the influence of lighting and environmental changes. By fusing IMU pre-integration with matching localization methods, the system can better cope with errors in dynamic environments and motion, improving the stability and reliability of localization. Furthermore, by combining the detection of point and line segment features, the system significantly enhances its adaptability to different environmental conditions, overcoming the limitations of traditional methods and providing higher performance and accuracy.

[0173] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0174] Based on the same inventive concept, this application also provides a map creation apparatus for implementing the map creation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more map creation apparatus embodiments provided below can be found in the limitations of the map creation method described above, and will not be repeated here.

[0175] In one embodiment, such as Figure 7 As shown, a map creation device is provided, including: an image acquisition module 701, a key point matching acquisition module 702, a line matching acquisition module 703, and a scene map construction module 704, wherein:

[0176] The image acquisition module 701 is used to acquire a first image and a second image captured by the binocular camera of the target scene.

[0177] The key point matching acquisition module 702 is used to input the first captured image and the second captured image into a pre-trained point and line detection model, and to acquire the first key point and the first line contained in the first captured image and the second key point and the second line contained in the second captured image through the point and line detection model. Based on the state information of the first key point and the second key point, the key point matching information of the first captured image and the second captured image is acquired.

[0178] The line matching acquisition module 703 is used to obtain line matching information between the first captured image and the second captured image based on key point matching information, the line parameters of the first line and the second line;

[0179] The scene map construction module 704 is used to obtain the three-dimensional coordinates of the scene lines contained in the target scene based on the line matching information, obtain the camera pose of the stereo camera based on the three-dimensional coordinates, and generate a scene map for the target scene using the camera pose.

[0180] In one embodiment, the line matching acquisition module 703 is further configured to acquire coordinate information of each first key point and second key point, and line endpoint information of each first line and second line; acquire point-line matching information based on coordinate information, line endpoint information and line parameters; the point-line matching information is used to characterize the matching relationship between the first key point and the first line, and the matching relationship between the second key point and the second line; and obtain line matching information based on the point-line matching information and the key point matching information.

[0181] In one embodiment, the coordinate information includes horizontal coordinate information and vertical coordinate information, and the line endpoint information includes the endpoint information of the first endpoint and the second endpoint, and the endpoint information includes the endpoint horizontal coordinate information and the endpoint vertical coordinate information; the line matching acquisition module 703 is further configured to obtain the distance between the current key point and the current line based on the coordinate information of the current key point and the line parameters of the current line; the current key point is either the first key point or the second key point, and the current line is either the first line or the second line; if the distance is less than a preset distance threshold, the endpoint information of the current first endpoint and the current second endpoint is obtained; the current first endpoint is the first endpoint of the current line, and the current second endpoint is the second endpoint of the current line; if the horizontal coordinate information of the current key point is located between the endpoint horizontal coordinate information of the current first endpoint and the current second endpoint, or if the vertical coordinate information of the current key point is located between the endpoint vertical coordinate information of the current first endpoint and the current second endpoint, point-line matching information between the current key point and the current line is generated.

[0182] In one embodiment, the line matching acquisition module 703 is further configured to acquire a first current line and a second current line; the first current line is any one of the first lines, and the second current line is any one of the second lines; based on the point-line matching information, acquire a first key point that matches the first current line and a second key point that matches the second current line; based on the key point matching information, acquire the number of matching key points between the first key point that matches the first current line and the second key point that matches the second current line; and generate line matching information between the first current line and the second current line when the number of matching key points meets a preset condition.

[0183] In one embodiment, the line matching acquisition module 703 is further configured to acquire the confidence level between the first current line and the second current line based on the number of matching key points; and generate line matching information between the first current line and the second current line when the number of matching key points is greater than a preset number threshold and the confidence level is greater than a preset confidence threshold.

[0184] In one embodiment, the key point matching acquisition module 702 is further configured to input the first captured image and the second captured image into the point and line detection model, obtain the first key point and the second key point through the key point detection branch of the point and line detection model, and obtain the first line and the second line through the line detection branch of the point and line detection model.

[0185] In one embodiment, the key point matching acquisition module 702 is further configured to acquire, through the key point detection branch, each image feature point contained in the first captured image and the second captured image, and the state information of each image feature point; acquire the self-attention feature and bidirectional attention feature of each image feature point according to the state information of each image feature point; and acquire the first key point and the second key point from the image feature points according to the self-attention feature and bidirectional attention feature of each image feature point.

[0186] In one embodiment, the scene map construction module 704 is further configured to determine the scene lines contained in the target scene and the Plück coordinates of each scene line based on the line matching information, the first line and the second line; use the Plück coordinates as the three-dimensional coordinates of each scene line, and obtain the camera coordinates of the stereo camera based on the three-dimensional coordinates of each scene line; and obtain the camera pose of the stereo camera based on the camera coordinates of the stereo camera.

[0187] In one embodiment, the image acquisition module 701 is further configured to acquire a first initial captured image and a second initial captured image captured by a binocular camera for the target scene, and acquire an event camera data stream captured by an event camera for the target scene; perform spatiotemporal alignment processing on the first initial captured image and the second initial captured image using the event camera data stream to obtain aligned first initial captured images and second initial captured images; input the aligned first initial captured images and the second initial captured images into a pre-trained dynamic region segmentation model to remove dynamic regions from the aligned first initial captured images and the second initial captured images to obtain the first captured image and the second captured image.

[0188] In one embodiment, the number of first initial captured images and second initial captured images captured by the binocular camera for the target scene is multiple, each corresponding to a different camera capture frame; the captured image acquisition module 701 is further configured to acquire the current first initial captured image and the current second initial captured image corresponding to the current camera capture frame; based on the timestamp corresponding to the current camera capture frame, acquire the current event data matching the timestamp from the event camera data stream, and the first event coordinates corresponding to the current event data in the first coordinate system; the first coordinate system is the coordinate system of the event camera; convert the first event coordinates into the second event coordinates corresponding to the current event data in the second coordinate system, and superimpose the second event coordinates onto the current first initial captured image and the current second initial captured image to obtain the aligned first initial captured image and the second initial captured image corresponding to the current camera capture frame; the second coordinate system is the coordinate system of the binocular camera.

[0189] In one embodiment, the number of first and second images captured by the binocular camera for the target scene is multiple, each corresponding to a different camera frame; the scene map construction module 704 is further used to obtain the target frame from the multiple camera frames and obtain the camera coordinates corresponding to each target frame; and obtain the camera pose of the binocular camera based on each camera coordinate.

[0190] In one embodiment, the scene map construction module 704 is further used to obtain the tracking features corresponding to the current shooting frame; the current shooting frame is any one of multiple camera shooting frames; the tracking features are key point matching information and line matching information corresponding to the current shooting frame; when the number of tracking features reaches a preset feature number threshold, and the degree of view difference between the first shooting image and the second shooting image corresponding to the current shooting frame does not reach a preset difference degree threshold, the current shooting frame is used as the target shooting frame.

[0191] Each module in the aforementioned map creation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the visual odometry in hardware form or independent of it, or stored in the memory of the visual odometry in software form, so that the processor can call and execute the corresponding operations of each module.

[0192] In one embodiment, a visual odometry is provided, which can be a terminal, and its internal structure diagram can be as follows: Figure 8As shown, the visual odometer includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a map creation method. The display unit of the visual odometer forms a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the visual odometer can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the visual odometer housing, or an external keyboard, touchpad, or mouse, etc.

[0193] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the visual odometry applied thereto. A specific visual odometry may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0194] In one embodiment, a visual odometry is also provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0195] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0196] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0197] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0198] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0199] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0200] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A map creation method, characterized in that, The method includes: Acquire the first and second images captured by the binocular camera of the target scene; The first captured image and the second captured image are input into a pre-trained point and line detection model. The first key point and the first line contained in the first captured image and the second key point and the second line contained in the second captured image are obtained through the point and line detection model. Based on the state information of the first key point and the second key point, the key point matching information of the first captured image and the second captured image is obtained. Based on the key point matching information, the line parameters of the first line and the second line, line matching information between the first captured image and the second captured image is obtained; Based on the line matching information, the three-dimensional coordinates of the scene lines contained in the target scene are obtained, and based on the three-dimensional coordinates, the camera pose of the stereo camera is obtained, and a scene map for the target scene is generated using the camera pose.

2. The method according to claim 1, characterized in that, The step of obtaining line matching information between the first captured image and the second captured image based on the key point matching information, the line parameters of the first line and the second line includes: Obtain the coordinate information of each of the first key points and the second key points, as well as the line endpoint information of each of the first lines and the second lines; Based on the coordinate information, the line endpoint information, and the line parameters, point-line matching information is obtained; the point-line matching information is used to characterize the matching relationship between the first key point and the first line, and the matching relationship between the second key point and the second line. The line matching information is obtained based on the point-line matching information and the key point matching information.

3. The method according to claim 2, characterized in that, The coordinate information includes horizontal coordinate information and vertical coordinate information; the line endpoint information includes the endpoint information of the first endpoint and the second endpoint; the endpoint information includes the endpoint horizontal coordinate information and the endpoint vertical coordinate information. The step of obtaining point-line matching information based on the coordinate information, the line endpoint information, and the line parameters includes: Based on the coordinate information of the current key point and the line parameters of the current line, the distance between the current key point and the current line is obtained; the current key point is either the first key point or the second key point, and the current line is either the first line or the second line. If the distance is less than a preset distance threshold, obtain the endpoint information of the current first endpoint and the current second endpoint; the current first endpoint is the first endpoint of the current line, and the current second endpoint is the second endpoint of the current line; If the horizontal coordinate information of the current key point is located between the horizontal coordinate information of the current first endpoint and the current second endpoint, or if the vertical coordinate information of the current key point is located between the vertical coordinate information of the current first endpoint and the current second endpoint, point-line matching information between the current key point and the current line is generated.

4. The method according to claim 2, characterized in that, The step of obtaining the line matching information based on the point-line matching information and the key point matching information includes: Obtain the first current line and the second current line; the first current line is any one of the first lines, and the second current line is any one of the second lines; Based on the point-line matching information, obtain the first key point that matches the first current line and the second key point that matches the second current line; Based on the key point matching information, obtain the number of matching key points between the first key point matched by the first current line and the second key point matched by the second current line; When the number of matching key points meets a preset condition, line matching information between the first current line and the second current line is generated.

5. The method according to claim 4, characterized in that, When the number of matching key points meets a preset condition, generating line matching information between the first current line and the second current line includes: Based on the number of matching key points, obtain the confidence level between the first current line and the second current line; If the number of matching key points is greater than a preset threshold and the confidence level is greater than a preset confidence level threshold, line matching information between the first current line and the second current line is generated.

6. The method according to claim 1, characterized in that, The acquisition of the first and second images captured by the binocular camera of the target scene includes: Acquire a first initial image and a second initial image captured by the binocular camera for the target scene, and acquire an event camera data stream captured by the event camera for the target scene; The first initial captured image and the second initial captured image are spatiotemporally aligned using the event camera data stream to obtain the aligned first initial captured image and the second initial captured image. The aligned first initial captured image and the second initial captured image are input into a pre-trained dynamic region segmentation model to remove dynamic regions from the aligned first initial captured image and the second initial captured image to obtain the first captured image and the second captured image.

7. The method according to claim 6, characterized in that, The binocular camera captures multiple first and second initial images of the target scene, each corresponding to a different camera frame. The process of performing spatiotemporal alignment processing on the first and second initial images using the event camera data stream to obtain aligned first and second initial images includes: Obtain the first initial captured image and the second initial captured image corresponding to the current camera frame; Based on the timestamp corresponding to the current camera frame, current event data matching the timestamp is obtained from the event camera data stream, and the current event data corresponds to the first event coordinates in the first coordinate system; the first coordinate system is the coordinate system of the event camera. The first event coordinates are converted into second event coordinates corresponding to the current event data in the second coordinate system, and the second event coordinates are superimposed on the current first initial captured image and the current second initial captured image to obtain the aligned first initial captured image and second initial captured image corresponding to the current camera captured frame; the second coordinate system is the coordinate system of the stereo camera.

8. The method according to claim 1, characterized in that, The binocular camera captures multiple first and second images of the target scene, each corresponding to a different camera frame. The process of obtaining the camera pose of the binocular camera based on the three-dimensional coordinates includes: Obtain the target frame from multiple camera capture frames, and obtain the camera coordinates corresponding to each target frame; The camera pose of the stereo camera is obtained based on the coordinates of each camera.

9. The method according to claim 8, characterized in that, The step of obtaining the target frame from multiple camera frames includes: Obtain the tracking features corresponding to the current shooting frame; the current shooting frame is any one of the multiple camera shooting frames; the tracking features are key point matching information and line matching information corresponding to the current shooting frame; If the number of features of the tracked features reaches a preset feature number threshold, and the difference in perspective between the first and second captured images corresponding to the current captured frame does not reach a preset difference threshold, then the current captured frame is taken as the target captured frame.

10. A map creation device, characterized in that, The device includes: The image acquisition module is used to acquire the first and second images captured by the binocular camera of the target scene; The key point matching acquisition module is used to input the first captured image and the second captured image into a pre-trained point and line detection model, obtain the first key point and the first line contained in the first captured image and the second key point and the second line contained in the second captured image through the point and line detection model, and obtain the key point matching information of the first captured image and the second captured image based on the state information of the first key point and the second key point. The line matching acquisition module is used to obtain line matching information between the first captured image and the second captured image based on the key point matching information, the line parameters of the first line and the second line; The scene map construction module is used to obtain the three-dimensional coordinates of the scene lines contained in the target scene according to the line matching information, obtain the camera pose of the stereo camera based on the three-dimensional coordinates, and generate a scene map for the target scene using the camera pose.

Images