Three-dimensional map construction method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a three-dimensional map construction method and device, equipment and a computer readable storage medium. The three-dimensional map construction method comprises the following steps: determining a relative pose transformation matrix between a video frame image and a reference video frame according to obtained feature points of the video frame image and matched feature points corresponding to the feature points in the reference video frame; the relative pose transformation matrix has relative scale information; based on the corresponding relative pose transformation matrix and the initial pose transformation matrix between the video frame image and the reference video frame, determining a corresponding updated pose transformation matrix between the video frame image and the reference video frame; the updated pose transformation matrix has absolute scale information; on the basis of the updated pose transformation matrix, pose information of feature points in the video frame image in the camera coordinate system is converted, a three-dimensional point cloud map is obtained, and the three-dimensional point cloud map has real scale information. According to the scheme, the accuracy of updating the pose transformation matrix can be improved, and a more accurate three-dimensional point cloud map can be obtained.

Description

Technical Field

[0001] The present application relates to the field of visual robot technology, and in particular to a three-dimensional map construction method, device, equipment and computer-readable storage medium. Background Art

[0002] As the application scope of technologies such as augmented reality (AR), visual SLAM (simultaneous localization and mapping), video projection, virtual-real registration and pose measurement becomes wider and wider, the processing requirements for pose data required to realize the above technologies are also getting higher and higher.

[0003] When performing posture estimation for visual SLAM, the posture transformation matrix between adjacent video frames cannot obtain the absolute scale information of the camera posture. Therefore, the three-dimensional point cloud data generated based on the posture transformation matrix also lacks scale information. Summary of the invention

[0004] The present application at least provides a three-dimensional map construction method, device, equipment and computer-readable storage medium.

[0005] In a first aspect, the present application provides a three-dimensional map construction method, the three-dimensional map construction method comprising:

[0006] Determine a relative posture transformation matrix between the video frame image and the reference video frame according to the feature points of the acquired video frame image and the matching feature points corresponding to the feature points in the reference video frame; the reference video frame is a video frame before or after the video frame image; the relative posture transformation matrix has relative scale information;

[0007] Based on the relative posture transformation matrix and the initial posture transformation matrix corresponding to the video frame image and the reference video frame, an updated posture transformation matrix corresponding to the video frame image and the reference video frame is determined; the initial posture transformation matrix is ​​determined based on the posture information in the offline map when the video frame image is collected and when the reference video frame is collected; the updated posture transformation matrix has absolute scale information;

[0008] Based on updating the pose transformation matrix, the pose information of the feature points in the video frame image in the camera coordinate system is transformed to obtain a three-dimensional point cloud map, which has real scale information.

[0009] Therefore, in this embodiment, a relative posture transformation matrix with relative scale information is obtained based on the feature points of the video frame image and the matching feature points in the reference video frame, and then the low-precision initial posture transformation matrix is ​​corrected based on the relative posture transformation matrix to obtain a high-precision updated posture transformation matrix with absolute scale information, and the feature points in the video frame image are transformed based on the updated posture transformation matrix with absolute scale information to obtain a three-dimensional point cloud map with real scale information. Absolute scale information is obtained by combining the relative posture transformation matrix with the low-precision initial posture transformation matrix, which improves the accuracy of the updated posture transformation matrix, and then the position information of the feature points in the video frame image with relative scale information is restored to position information with real scale information based on the updated posture transformation matrix with absolute scale information, thereby obtaining a more accurate three-dimensional point cloud map.

[0010] In some embodiments, the relative posture transformation matrix includes a relative rotation transformation matrix and a relative translation transformation matrix; the initial posture transformation matrix includes an initial translation transformation matrix;

[0011] Based on the relative pose transformation matrix and the initial pose transformation matrix corresponding to the video frame image and the reference video frame, determining the updated pose transformation matrix corresponding to the video frame image and the reference video frame, including:

[0012] Based on the relative translation transformation matrix and the initial translation transformation matrix corresponding to the video frame image and the reference video frame, an updated posture transformation matrix corresponding to the video frame image and the reference video frame is determined.

[0013] In this embodiment, since the real scale information of the three-dimensional point cloud map is only related to the translation transformation matrix, the updated posture transformation matrix with absolute scale information can be obtained based on the relative translation transformation matrix and the initial translation transformation matrix.

[0014] In some embodiments, determining an updated posture transformation matrix corresponding to the video frame image and the reference video frame based on the relative translation transformation matrix and the initial translation transformation matrix corresponding to the video frame image and the reference video frame includes:

[0015] The absolute value of the initial translation transformation matrix corresponding to the video frame image and the reference video frame is used as the absolute scale parameter;

[0016] Based on the relative translation transformation matrix and the absolute scale parameter corresponding to the video frame image and the reference video frame, an updated translation transformation matrix is ​​obtained; the updated translation transformation matrix has absolute scale information;

[0017] The updated translation transformation matrix and the relative rotation transformation matrix are used as the updated pose transformation matrix.

[0018] Therefore, in this embodiment, the absolute scale parameter is obtained based on the absolute value of the initial translation transformation matrix, and then the updated translation transformation matrix is ​​determined according to the absolute scale parameter and the relative translation transformation matrix, and the updated translation transformation matrix and the relative rotation transformation matrix are used as the updated posture transformation matrix with higher accuracy.

[0019] In some embodiments, based on the relative translation transformation matrix and the absolute scale parameter corresponding to the video frame image and the reference video frame, obtaining an updated translation transformation matrix includes:

[0020] Based on the product of the relative translation transformation matrix corresponding to the video frame image and the reference video frame and the absolute scale parameter, an updated translation transformation matrix is ​​obtained.

[0021] Therefore, in this embodiment, based on the product of the relative translation transformation matrix and the absolute scale parameter, an updated translation transformation matrix with higher accuracy is obtained.

[0022] In some embodiments, determining a relative posture transformation matrix between the video frame image and the reference video frame according to the acquired feature points of the video frame image and the matching feature points corresponding to the feature points in the reference video frame includes:

[0023] Obtain the video frame image captured by the camera, and extract feature points in the video frame image;

[0024] Using a feature point matching algorithm to determine matching feature points in the reference video frame corresponding to feature points in the video frame image, and obtain feature point matching pairs between the video frame image and the reference video frame;

[0025] The relative pose transformation matrix is ​​determined based on the feature point matching pairs between the video frame image and the reference video frame.

[0026] Therefore, in this embodiment, a feature point matching pair is formed based on the feature points in the video frame image and the matching feature points in the reference video frame, and then a relative posture transformation matrix with relative scale information is calculated based on the posture information of the feature points in the feature point matching pair and the posture information of the matching feature points.

[0027] In some embodiments, before the step of converting the pose information of the feature points in the video frame image in the camera coordinate system based on the updated pose transformation matrix to obtain the three-dimensional point cloud map, the step further includes:

[0028] Based on the updated posture transformation matrix, the posture of the feature points in the video frame image is transformed to obtain the corresponding projection points of the feature points in the reference video frame;

[0029] Based on the projection points corresponding to the feature points and the matching feature points, determine whether to output the updated pose transformation matrix.

[0030] Therefore, in this embodiment, the position information of the feature points in the video frame image is projected into the reference video frame based on the updated pose transformation matrix, and the updated pose transformation matrix is ​​verified based on the position information of the projection points and the matching feature points in the reference video frame to improve the accuracy of the updated pose transformation matrix.

[0031] In some embodiments, the pose information of feature points in the video frame image in the camera coordinate system is converted based on the updated pose transformation matrix to obtain a three-dimensional point cloud map, including:

[0032] In response to the coincidence of the projection point corresponding to the feature point with the matching feature point, the position information of the feature point in the video frame image in the camera coordinate system is converted to the coordinate system of the offline map based on the updated posture transformation matrix to obtain a three-dimensional point cloud map.

[0033] Therefore, in this embodiment, when the projection point corresponding to the feature point coincides with the matching feature point, it is determined that the updated pose transformation matrix is ​​valid, and the position information of the feature point in the video frame image in the camera coordinate system can be transformed based on the updated pose transformation matrix to obtain a three-dimensional point cloud map.

[0034] A second aspect of the present application provides a three-dimensional map construction device, the three-dimensional map construction device comprising:

[0035] A calculation module, used to determine a relative posture transformation matrix between the video frame image and a reference video frame according to the acquired feature points of the video frame image and the matching feature points corresponding to the feature points in the reference video frame; the reference video frame is a video frame before or after the video frame image;

[0036] An analysis module is used to determine an updated posture transformation matrix corresponding to the video frame image and the reference video frame based on a relative posture transformation matrix and an initial posture transformation matrix corresponding to the video frame image and the reference video frame; the initial posture transformation matrix is ​​determined based on the posture information of the feature points in the video frame image and the matching feature points of the reference video frame in the offline map respectively;

[0037] The construction module is used to transform the pose information of the feature points in the video frame image in the camera coordinate system based on the updated pose transformation matrix to obtain a three-dimensional point cloud map.

[0038] Therefore, in this embodiment, a relative posture transformation matrix with relative scale information is obtained based on the feature points of the video frame image and the matching feature points in the reference video frame, and then the low-precision initial posture transformation matrix is ​​corrected based on the relative posture transformation matrix to obtain a high-precision updated posture transformation matrix with absolute scale information, and the feature points in the video frame image are transformed based on the updated posture transformation matrix with absolute scale information to obtain a three-dimensional point cloud map with real scale information. Absolute scale information is obtained by combining the relative posture transformation matrix with the low-precision initial posture transformation matrix, which improves the accuracy of the updated posture transformation matrix, and then the position information of the feature points in the video frame image with relative scale information is restored to position information with real scale information based on the updated posture transformation matrix with absolute scale information, thereby obtaining a more accurate three-dimensional point cloud map.

[0039] A third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the three-dimensional map construction method in the first aspect.

[0040] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implements the three-dimensional map construction method in the first aspect.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0043] Figure 1 It is a flowchart of the three-dimensional map construction method provided by this application;

[0044] Figure 2 yes Figure 1 A schematic diagram of a specific embodiment of step S1 in the provided three-dimensional map construction method;

[0045] Figure 3 It is a schematic diagram of the framework of an embodiment of a three-dimensional map construction device provided by the present application;

[0046] Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0047] Figure 5 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0048] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0049] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0050] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0051] See also Figure 1 , Figure 1 It is a flowchart of the three-dimensional map construction method provided by this application.

[0052] This embodiment provides a three-dimensional map construction method. The execution subject of the three-dimensional map construction method can be an image processing device. The image processing device can be any terminal device or server or other processing device that can execute the method embodiment of the present application, wherein the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The visual positioning method provided in this embodiment can be applied to a processing device mounted on a mobile device such as a robot, a drone, and a car. Specifically, the three-dimensional map construction method may include the following steps.

[0053] Step S1: Determine the relative posture transformation matrix between the video frame image and the reference video frame according to the feature points of the acquired video frame image and the matching feature points corresponding to the feature points in the reference video frame; the reference video frame is a video frame before or after the video frame image, and the relative posture transformation matrix has relative scale information.

[0054] Step S2: Based on the relative pose transformation matrix and the initial pose transformation matrix corresponding to the video frame image and the reference video frame, determine the updated pose transformation matrix corresponding to the video frame image and the reference video frame; the initial pose transformation matrix is ​​determined based on the pose information in the offline map when the video frame image is collected and when the reference video frame is collected; the updated pose transformation matrix has absolute scale information.

[0055] Step S3: Based on the updated posture transformation matrix, the posture information of the feature points in the video frame image in the camera coordinate system is transformed to obtain a three-dimensional point cloud map, and the three-dimensional point cloud map has real scale information.

[0056] Specifically, the specific steps of determining the relative posture transformation matrix between the video frame image and the reference video frame in step S1 are as follows.

[0057] See also Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific embodiment of step S1 in the provided three-dimensional map construction method.

[0058] S11: Obtain a video frame image captured by a camera, and extract feature points in the video frame image.

[0059] In this embodiment, feature points can be simply understood as more prominent points in the video frame image, such as contour points, bright spots in darker areas, dark spots in brighter areas, etc. In one embodiment, the camera used to capture the video frame image can be a monocular camera but not limited to a binocular camera. Feature points of the video frame image can be extracted based on SIFT (scale invariant feature transform) feature points, SURF (Speeded Up Robust Features) feature points and ORB (Oriented FAST and Rotated BRIEF) feature point extraction algorithms, or feature points of the video frame image can be extracted using a trained neural network. Each feature point has a descriptor that indicates a unique identity and characteristics.

[0060] Specifically, the position coordinates, angle and other information of each feature point in the video frame are associated with the posture of the camera. Therefore, each feature point in the video frame has posture information in the camera coordinate system.

[0061] S12: using a feature point matching algorithm to determine matching feature points in the reference video frame corresponding to the feature points in the video frame image, and obtaining feature point matching pairs between the video frame image and the reference video frame.

[0062] In this embodiment, two similar descriptors can be matched by calculating the Euclidean distance, Hamming distance, etc. between the descriptors of the two feature points. Specifically, the Euclidean distance between the two descriptors is calculated. The feature points of this matching method include SIFT feature points, SURF feature points, feature points extracted based on neural networks, etc. The Hamming distance between the two descriptors is calculated. The feature points of this matching method include ORB feature points, BRISK feature points, feature points extracted based on neural networks, etc.

[0063] In a specific embodiment, the feature point matching algorithm includes but is not limited to a Brute-FroceMatcher method.

[0064] Based on the feature point matching algorithm, the distance between the feature points in the video frame image and the feature points in the reference video frame is calculated, and then the matching feature points in the reference video frame that match the feature points in the video frame image are determined, and then feature point matching pairs are formed based on the feature points in the video frame image and the corresponding matching feature points in the reference video frame. The reference video frame is a video frame before or after the video frame image. Specifically, the reference video frame is a video frame before or after the video frame image that is adjacent to the video frame image. In this embodiment, the reference video frame is a video frame after the video frame image and adjacent to the video frame image.

[0065] S13: Determine a relative posture transformation matrix based on feature point matching pairs between the video frame image and the reference video frame.

[0066] Specifically, according to the correspondence between the pose information of the feature points in the feature point matching pair and the matching feature points in the camera coordinate system, the relative pose transformation matrix corresponding to the video frame image and the reference video frame is obtained.

[0067] In this embodiment, the relative posture transformation matrix includes the relative rotation transformation matrix dR c and the relative translation transformation matrix dT c . In order to unify the dimensions, the relative translation transformation matrix is ​​a normalized vector. The relative posture transformation matrix has relative scale information, which refers to the scale information in the image coordinate system, that is, the scale of the parameters in the posture transformation matrix is ​​the scale in the image coordinate system.

[0068] Specifically, the specific steps of determining the corresponding updated pose transformation matrix between the video frame image and the reference video frame in step S2 are as follows.

[0069] In one embodiment, in a specific embodiment, the pose information of the camera in the world coordinate system when capturing video frame images and the position information of the camera in the world coordinate system when capturing reference video frames are obtained through a global navigation satellite system (GNSS).

[0070] The initial posture transformation matrix is ​​determined based on the posture information in the offline map when the video frame image is collected and the reference video frame is collected. In this embodiment, the offline map is a world coordinate system. That is to say, the initial posture transformation matrix between the video frame image and the reference video frame is obtained based on the correspondence between the posture information of the camera in the world coordinate system when the video frame image is collected and the position information of the camera in the world coordinate system when the reference video frame is collected. In this embodiment, the accuracy of the initial posture transformation matrix is ​​low. Among them, the initial posture transformation matrix includes the initial rotation transformation matrix dR prior and the initial translation transformation matrix dT prior .

[0071] In one embodiment, based on the corresponding relative translation transformation matrix dT between the video frame image and the reference video frame c and the initial translation transformation matrix dT prior , determine the corresponding updated pose transformation matrix between the video frame image and the reference video frame.

[0072] In a specific embodiment, the absolute value of the initial translation transformation matrix between the video frame image and the reference video frame is used as the absolute scale parameter s, that is, s=|dT prior |. Based on the relative translation transformation matrix corresponding to the video frame image and the reference video frame and the absolute scale parameter, an updated translation transformation matrix is ​​obtained; the updated translation transformation matrix has absolute scale information. Specifically, the updated translation transformation matrix is ​​obtained based on the product of the relative translation transformation matrix corresponding to the video frame image and the reference video frame and the absolute scale parameter. Right now The translation transformation matrix will be updated and the relative rotation transformation matrix dR c As the updated pose transformation matrix. The absolute scale information parameter represents the scale information in the world coordinate system. The updated translation transformation matrix has the absolute scale information, that is, the scale of the parameters in the updated translation transformation matrix is ​​the scale in the world coordinate system.

[0073] Specifically, the specific steps of obtaining the three-dimensional point cloud map in step S3 are as follows.

[0074] In one embodiment, the pose transformation of feature points in the video frame image is performed based on the updated pose transformation matrix to obtain the projection points corresponding to the feature points in the reference video frame; based on the projection points corresponding to the feature points and the matching feature points, it is determined whether to output the updated pose transformation matrix.

[0075] Specifically, in response to the position error between the projection point corresponding to the feature point in the reference video frame and the matching feature point being less than the error threshold, it is determined that the projection point coincides with the matching feature point, and the corresponding updated pose transformation matrix is ​​output. In response to the position error between the projection point corresponding to the feature point in the reference video frame and the matching feature point being not less than the error threshold, the error between the task projection point and the matching feature point is large, and the updated pose transformation matrix between the video frame image and the reference video frame is recalculated.

[0076] In a specific embodiment, in response to the projection point corresponding to the feature point coinciding with the matching feature point, the position information of the feature point in the video frame image in the camera coordinate system is converted to the coordinate system of the offline map based on the updated posture transformation matrix to obtain a three-dimensional point cloud map.

[0077] Specifically, the three-dimensional coordinates corresponding to the video frame image can be determined according to the updated posture transformation matrix. The three-dimensional coordinates corresponding to the video frame image refer to the three-dimensional coordinates corresponding to the camera. After determining the posture transformation matrix between the video frame image and the reference image, the three-dimensional coordinates of any video frame image can be calculated according to the transformation relationship. The three-dimensional coordinates corresponding to the video frame image actually refer to the three-dimensional point coordinates corresponding to the position when the camera took the video frame image.

[0078] Since the three-dimensional coordinates corresponding to each video frame image are the three-dimensional coordinates in the corresponding camera coordinate system, the coordinates of the feature points in the video frame image are also in the camera coordinate system. In order to convert the coordinates of the feature points to the world coordinate system, the three-dimensional coordinates of the feature points in the world coordinate system are obtained according to the updated posture transformation matrix, thereby obtaining a three-dimensional point cloud map with real scale information.

[0079] The three-dimensional map construction method provided in this embodiment obtains a relative posture transformation matrix with relative scale information based on the feature points of the video frame image and the matching feature points in the reference video frame, and then corrects the low-precision initial posture transformation matrix based on the relative posture transformation matrix to obtain a high-precision updated posture transformation matrix with absolute scale information, and transforms the feature points in the video frame image based on the updated posture transformation matrix with absolute scale information to obtain a three-dimensional point cloud map with real scale information. Absolute scale information is obtained by combining the relative posture transformation matrix with the low-precision initial posture transformation matrix, which improves the accuracy of the updated posture transformation matrix, and then the position information of the feature points in the video frame image with relative scale information is restored to position information with real scale information based on the updated posture transformation matrix with absolute scale information, thereby obtaining a more accurate three-dimensional point cloud map.

[0080] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0081] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of a three-dimensional map construction device provided by the present application.

[0082] This embodiment provides a three-dimensional map construction device 60 , which includes a calculation module 61 , an analysis module 62 and a construction module 63 .

[0083] The calculation module 61 is used to determine the relative posture transformation matrix between the video frame image and the reference video frame based on the feature points of the acquired video frame image and the matching feature points corresponding to the feature points in the reference video frame; the reference video frame is the video frame before or after the video frame image.

[0084] The analysis module 62 is used to determine the updated pose transformation matrix corresponding to the video frame image and the reference video frame based on the relative pose transformation matrix and the initial pose transformation matrix corresponding to the video frame image and the reference video frame; the initial pose transformation matrix is ​​determined based on the pose information of the feature points in the video frame image and the matching feature points of the reference video frame in the offline map.

[0085] The construction module 63 is used to transform the pose information of the feature points in the video frame image in the camera coordinate system based on the updated pose transformation matrix to obtain a three-dimensional point cloud map.

[0086] The three-dimensional map construction device provided in this embodiment obtains a relative posture transformation matrix with relative scale information based on the feature points of the video frame image and the matching feature points in the reference video frame, and then corrects the low-precision initial posture transformation matrix based on the relative posture transformation matrix to obtain a high-precision updated posture transformation matrix with absolute scale information, and transforms the feature points in the video frame image based on the updated posture transformation matrix with absolute scale information to obtain a three-dimensional point cloud map with real scale information. Absolute scale information is obtained by combining the relative posture transformation matrix with the low-precision initial posture transformation matrix, which improves the accuracy of the updated posture transformation matrix, and then the position information of the feature points in the video frame image with relative scale information is restored to position information with real scale information based on the updated posture transformation matrix with absolute scale information, thereby obtaining a more accurate three-dimensional point cloud map.

[0087] See also Figure 4 , Figure 4 80 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-mentioned three-dimensional map construction method embodiments. In a specific implementation scenario, the electronic device 80 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 80 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0088] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps in any of the above-mentioned three-dimensional map construction method embodiments. The processor 82 can also be called a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0089] See also Figure 5 , Figure 5 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

[0090] The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor, and the program instructions 901 are used to implement the steps in any of the above-mentioned three-dimensional map construction method embodiments.

[0091] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0092] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0093] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0094] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0095] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0096] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A three-dimensional map construction method, characterized in that: include: Determine a relative posture transformation matrix between the video frame image and the reference video frame according to the feature points of the acquired video frame image and the matching feature points corresponding to the feature points in the reference video frame; the reference video frame is a video frame before or after the video frame image; the relative posture transformation matrix has relative scale information; Based on the relative posture transformation matrix and the initial posture transformation matrix corresponding to the video frame image and the reference video frame, determining the updated posture transformation matrix corresponding to the video frame image and the reference video frame; the initial posture transformation matrix is ​​determined based on the posture information in the offline map when the video frame image is collected and when the reference video frame is collected; the updated posture transformation matrix has absolute scale information; Based on the updated pose transformation matrix, the pose information of the feature points in the video frame image in the camera coordinate system is transformed to obtain a three-dimensional point cloud map, wherein the three-dimensional point cloud map has real scale information.

2. The three-dimensional map construction method according to claim 1, characterized in that: The relative posture transformation matrix includes a relative rotation transformation matrix and a relative translation transformation matrix; the initial posture transformation matrix includes an initial translation transformation matrix; The determining, based on the relative posture transformation matrix and the initial posture transformation matrix corresponding to the video frame image and the reference video frame, an updated posture transformation matrix corresponding to the video frame image and the reference video frame comprises: Based on the relative translation transformation matrix and the initial translation transformation matrix corresponding to the video frame image and the reference video frame, the updated posture transformation matrix corresponding to the video frame image and the reference video frame is determined.

3. The three-dimensional map construction method according to claim 2, characterized in that: The determining, based on the relative translation transformation matrix and the initial translation transformation matrix corresponding to the video frame image and the reference video frame, the updated posture transformation matrix corresponding to the video frame image and the reference video frame comprises: Taking the absolute value of the initial translation transformation matrix between the video frame image and the reference video frame as an absolute scale parameter; Based on the relative translation transformation matrix corresponding to the video frame image and the reference video frame and the absolute scale parameter, an updated translation transformation matrix is ​​obtained; the updated translation transformation matrix has absolute scale information; The updated translation transformation matrix and the relative rotation transformation matrix are used as the updated posture transformation matrix.

4. The three-dimensional map construction method according to claim 3, characterized in that: The obtaining of an updated translation transformation matrix based on the relative translation transformation matrix corresponding to the video frame image and the reference video frame and the absolute scale parameter comprises: The updated translation transformation matrix is ​​obtained based on the product of the relative translation transformation matrix corresponding to the video frame image and the reference video frame and the absolute scale parameter.

5. The three-dimensional map construction method according to any one of claims 1 to 4, characterized in that: The step of determining a relative posture transformation matrix between the video frame image and the reference video frame according to the acquired feature points of the video frame image and the matching feature points corresponding to the feature points in the reference video frame includes: Acquire the video frame image captured by a camera, and extract feature points in the video frame image; Determine, in the reference video frame, the matching feature points corresponding to the feature points in the video frame image using a feature point matching algorithm, and obtain a feature point matching pair between the video frame image and the reference video frame; The relative posture transformation matrix is ​​determined based on feature point matching pairs between the video frame image and the reference video frame.

6. The three-dimensional map construction method according to any one of claims 1 to 4, characterized in that: Before the step of converting the pose information of the feature points in the video frame image in the camera coordinate system based on the updated pose transformation matrix to obtain a three-dimensional point cloud map, the step further includes: Performing posture transformation on the feature points in the video frame image based on the updated posture transformation matrix to obtain projection points corresponding to the feature points in the reference video frame; Based on the projection point corresponding to the feature point and the matching feature point, it is determined whether to output the updated pose transformation matrix.

7. The three-dimensional map construction method according to claim 6, characterized in that: The step of converting the pose information of the feature points in the video frame image in the camera coordinate system based on the updated pose transformation matrix to obtain a three-dimensional point cloud map includes: In response to the projection point corresponding to the feature point coinciding with the matching feature point, the position information of the feature point in the video frame image in the camera coordinate system is converted to the coordinate system of the offline map based on the updated posture transformation matrix to obtain the three-dimensional point cloud map.

8. A three-dimensional map construction device, characterized in that: include: A calculation module, used to determine a relative posture transformation matrix between the video frame image and the reference video frame according to the feature points of the acquired video frame image and the matching feature points corresponding to the feature points in the reference video frame; the reference video frame is a video frame before or after the video frame image; the relative posture transformation matrix has relative scale information; An analysis module, configured to determine an updated posture transformation matrix corresponding to the video frame image and the reference video frame based on the relative posture transformation matrix and the initial posture transformation matrix corresponding to the video frame image and the reference video frame; the initial posture transformation matrix is ​​determined based on the posture information in the offline map when the video frame image is collected and when the reference video frame is collected; the updated posture transformation matrix has absolute scale information; A construction module is used to transform the pose information of the feature points in the video frame image in the camera coordinate system based on the updated pose transformation matrix to obtain a three-dimensional point cloud map, wherein the three-dimensional point cloud map has real scale information.

9. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the three-dimensional map construction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the three-dimensional map construction method described in any one of claims 1 to 7 is implemented.

Images