Data association method and device, server and storage medium

Abstract

The embodiment of the invention discloses a data association method and device, a server and a storage medium, and the method comprises the steps: obtaining two collection data, selecting one collection data as reference data, and taking the other collection data as to-be-associated data; obtaining three-dimensional point cloud data corresponding to the image feature points in the reference data by using a three-dimensional reconstruction technology; acquiring a first reference image and a first associated image; determining matched image feature points between the first reference image and the first associated image and a target pose of the first associated image; obtaining a coarse pose of the second associated image; determining matched image feature points between the second associated image and the second reference image; determining a target pose of the second associated image; and taking the second associated image as a new first associated image, taking the unprocessed associated image as a new second associated image, and returning to the step of obtaining the coarse pose of the second associated image for continuous execution. According to the technical scheme, the association of the image feature points can be efficiently and simply realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of high-precision maps, and in particular to a data association method, device, server and storage medium. Background Art

[0002] As electronic maps evolve from standard maps to high-precision maps, high-precision maps have begun to play an important role in many fields such as smart cities and smart driving. Since the solution based on laser data collection and construction of high-precision maps is costly and difficult to deploy on a large scale, the solution based on vision to build high-precision maps has become one of the important technical directions in the field of high-precision map construction due to its low cost and high efficiency. However, in the process of using visual solutions to build high-precision maps, since the images taken by the camera cannot directly obtain the three-dimensional position information of the real world, the visual solution needs to use multi-view images and the trajectory data collected when collecting images to perform geometric operations to obtain the three-dimensional position information of the ground features in the real world.

[0003] In order to make the accuracy of the three-dimensional position information of the road obtained by the visual solution meet the accuracy requirements of the high-precision map, it is necessary to align the trajectory data collected in multiple passes. The bundle adjustment method (BA) based on visual feature points is the most accurate trajectory alignment solution available, but the inventors of the present disclosure have found that the prerequisite for using this solution to align the trajectory data collected in multiple passes is that the image feature points of the images corresponding to the trajectory data of different passes need to be data associated. However, due to the problems of changes in lighting conditions, changes in viewing angles, large initial posture deviations, and high repetition of road elements, the matching of image feature points between the trajectory data collected in multiple passes becomes more difficult.

[0004] Therefore, it is necessary to provide a simple, efficient and reliable scheme for associating image feature points of an image so that trajectory data corresponding to the image can be aligned with high precision based on the bundle adjustment method of visual feature points. Summary of the invention

[0005] Embodiments of the present disclosure provide a data association method, device, server, and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a data association method, including:

[0007] Acquire two rounds of collected data on the same road, wherein each round of collected data includes: multiple frames of images collected on the road, image feature points of each frame of image, and initial position and relative position of each frame of image within the round;

[0008] From the two rounds of collected data, one round of collected data is selected as reference data, and the other round of collected data is selected as data to be associated;

[0009] Using three-dimensional reconstruction technology, obtain three-dimensional point cloud data corresponding to the image feature points in the reference data;

[0010] Acquire multiple frames of first reference images and first associated images having a common viewing relationship from the reference images included in the reference data and the associated images included in the to-be-associated data;

[0011] Determine the image feature points that match between the first reference image and the first associated image, and the target pose of the first associated image based on the initial pose, image feature points, and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, and the intra-pass relative pose of the first associated image, and the intrinsic parameters of the camera used to acquire the first associated image;

[0012] Obtaining a rough pose of a second associated image based on the target pose of the first associated image and the intra-pass relative poses of second associated images other than the first associated image in the to-be-associated data;

[0013] Determine image feature points that match between the second associated image and the second reference image based on image feature points, a rough pose, an intrinsic parameter of a camera for acquiring the second associated image, and an initial pose, image feature points, and the three-dimensional point cloud data corresponding to the image feature points of a second reference image in the reference data that has a co-viewing relationship with the second associated image;

[0014] If the to-be-associated data also includes an unprocessed associated image, determining a target pose of the second associated image based on image feature points matching the second associated image and the three-dimensional point cloud data corresponding to the image feature points;

[0015] The step of taking the second associated image as a new first associated image, taking the unprocessed associated image as a new second associated image, returning the target pose based on the first associated image and the intra-pass relative pose of the second associated image other than the first associated image in the data to be associated, and obtaining the rough pose of the second associated image is continued until all the associated images in the data to be associated are processed.

[0016] In a second aspect, an embodiment of the present invention provides a data association device, including:

[0017] The first acquisition module is configured to acquire two trips of collected data on the same road, wherein each trip of collected data includes: multiple frames of images collected on the road, image feature points of each frame of image, and an initial position and relative position of each frame of image within the trip;

[0018] A first determination module is configured to select one of the at least two acquisition data as reference data, and determine the three-dimensional point cloud data corresponding to the image feature points based on the initial pose of the image in the reference data and the image feature points in the image in the reference data;

[0019] A selection module is configured to select one of the two rounds of collected data as reference data and the other round of collected data as data to be associated;

[0020] A second determination module is configured to obtain three-dimensional point cloud data corresponding to the image feature points in the reference data using three-dimensional reconstruction technology;

[0021] A second acquisition module is configured to acquire a plurality of frames of first reference images and first associated images having a common viewing relationship from the reference images included in the reference data and the associated images included in the to-be-associated data;

[0022] A third determination module is configured to determine the image feature points that match between the first reference image and the first associated image, and the target pose of the first associated image based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, the intra-pass relative pose of the first associated image and the camera intrinsic parameters for acquiring the first associated image;

[0023] a fourth determination module, configured to obtain a rough pose of a second associated image based on the target pose of the first associated image and an intra-pass relative pose of a second associated image other than the first associated image in the to-be-associated data;

[0024] a fifth determination module, configured to determine image feature points that match between the second associated image and the second reference image based on image feature points, a coarse pose, an intrinsic parameter of a camera for acquiring the second associated image, an initial pose, an image feature point, and the three-dimensional point cloud data corresponding to the image feature point of a second reference image in the reference data that has a co-viewing relationship with the second associated image;

[0025] a sixth determination module, configured to determine a target pose of the second associated image based on image feature points matching the second associated image and the three-dimensional point cloud data corresponding to the image feature points if the to-be-associated data also includes an unprocessed associated image;

[0026] The seventh determination module is configured to use the second associated image as a new first associated image and the unprocessed associated image as a new second associated image, return the target posture based on the first associated image and the intra-pass relative posture of the second associated image other than the first associated image in the data to be associated, and continue to execute the step of obtaining the rough posture of the second associated image until all the associated images in the data to be associated are processed.

[0027] The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.

[0028] In one possible design, the structure of the above-mentioned device includes a memory and a processor, the memory is used to store one or more computer instructions that support the above-mentioned device to execute the above-mentioned corresponding method, and the processor is configured to execute the computer instructions stored in the memory. The above-mentioned device may also include a communication interface for the above-mentioned device to communicate with other devices or communication networks.

[0029] In a third aspect, an embodiment of the present disclosure provides a server, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in any one of the above aspects.

[0030] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by any of the above-mentioned devices, and when the computer instructions are executed by a processor, they are used to implement the method described in any of the above-mentioned aspects.

[0031] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which includes computer instructions, and when the computer instructions are executed by a processor, they are used to implement the method described in any of the above aspects.

[0032] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0033] The disclosed embodiment provides a data association method, which can select one of the collected data as reference data and the other collected data as data to be associated after acquiring two collected data of the same road. Using three-dimensional reconstruction technology, three-dimensional point cloud data corresponding to image feature points in the reference data is obtained. In order to align the pose of the data to be associated with the reference data, multiple frames of images can be selected from the data to be associated as first associated images, and multiple frames of first reference images having a co-viewing relationship with the first associated image can be selected from the reference data. Then, based on the initial pose of the first reference image, the image feature points on the first reference image and the three-dimensional point cloud data corresponding to the image feature points, the initial pose of the first associated image, the image feature points, the first relative pose of the first associated image, and the camera internal parameters of the first associated image, the image feature points that match between the first associated image and the first reference image and the target pose of the first associated image are determined after aligning the first associated image with the corresponding first reference image. Afterwards, the rough pose of the second associated image can be determined based on the target pose of the first associated image and the relative pose within the pass of the second associated image other than the first associated image in the data to be associated. The image feature points that match between the second associated image and the second reference image can be determined based on the image feature points, rough pose, camera internal parameters for collecting the second associated image, the initial pose of the second reference image in the reference data that has a co-viewing relationship with the second associated image, image feature points, and three-dimensional point cloud data corresponding to the image feature points. All associated images in the data to be associated are processed in the same manner as above, so that the reference data and the associated data can be preliminarily associated and aligned. The above method provided by the embodiment of the present disclosure can efficiently and simply realize the association of image feature points of images in two passes of collected data by first associating and aligning some images in the data to be associated with the reference data, and then associating and aligning other images in the data to be associated with the reference data based on the aligned partial images.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0036] Figure 1 A flow chart of a data association method according to an embodiment of the present disclosure is shown.

[0037] Figure 2 A schematic diagram showing a complete process implementation of a data association method according to an embodiment of the present disclosure is shown.

[0038] Figure 3 A structural block diagram of a data association device according to an embodiment of the present disclosure is shown.

[0039] Figure 4 It is a schematic diagram of the structure of a server suitable for implementing the data association method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0041] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0042] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] The user information (including but not limited to user device information such as location information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0044] The details of the embodiments of the present disclosure are described in detail below through specific examples.

[0045] Figure 1 FIG. 1 is a flow chart showing a data association method according to an embodiment of the present disclosure. Figure 1 As shown, the data association method includes the following steps:

[0046] In step S101, two trips of collected data on the same road are obtained, wherein each trip of collected data includes: multiple frames of images collected on the road, image feature points of each frame of image, and initial posture of each frame of image, and relative posture within the trip;

[0047] In step S102, one of the two rounds of collected data is selected as reference data, and the other round of collected data is selected as data to be associated;

[0048] In step S103, three-dimensional point cloud data corresponding to the image feature points in the reference data is obtained by using three-dimensional reconstruction technology;

[0049] In step S104, a plurality of frames of first reference images and first associated images having a common viewing relationship are acquired from the reference images included in the reference data and the associated images included in the to-be-associated data;

[0050] In step S105, based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, intra-pass relative pose of the first associated image and the intrinsic parameters of the camera used to acquire the first associated image, the image feature points that match between the first reference image and the first associated image, and the target pose of the first associated image are determined;

[0051] In step S106, a rough pose of the second associated image is obtained based on the target pose of the first associated image and the intra-pass relative poses of the second associated images other than the first associated image in the to-be-associated data;

[0052] In step S107, based on the image feature points and coarse pose of the second associated image, the intrinsic parameters of the camera for acquiring the second associated image, and the initial pose, image feature points, and the three-dimensional point cloud data corresponding to the image feature points of the second reference image in the reference data that has a co-viewing relationship with the second associated image, the image feature points matching the second associated image and the second reference image are determined;

[0053] In step S108, if the to-be-associated data also includes an unprocessed associated image, the target pose of the second associated image is determined based on the image feature points that match the second associated image and the three-dimensional point cloud data corresponding to the image feature points;

[0054] In step S109, the second associated image is used as a new first associated image, and the unprocessed associated image is used as a new second associated image. The step of returning the target posture based on the first associated image and the intra-pass relative posture of the second associated image other than the first associated image in the data to be associated is continued to obtain the rough posture of the second associated image until all the associated images in the data to be associated are processed.

[0055] In this embodiment, the data association method can be executed on a server or on a vehicle-mounted device, and the present disclosure does not impose any restrictions. The data association method is applicable to the scenario of associating image feature points of images in two passes of collected data during the construction of a high-precision map based on vision, and the association results of the image feature points can be used to align the trajectory data corresponding to the image, and the alignment scheme can be a bundle adjustment method based on visual feature points.

[0056] For ease of understanding, the present disclosure briefly introduces data collection. The data collection vehicle can perform multiple trips of data collection on any road in the real world. The collected data usually includes trajectory data and image data, wherein the trajectory data is composed of a series of positioning points formed when the data collection vehicle travels on the road. The positioning technology can adopt multi-source fusion positioning, such as inertial navigation, GNSS, etc. When collecting trajectory data, the image sensing device loaded on the data collection vehicle, such as a camera, can take an image of the road. The image and trajectory data can be matched by the collection time. After the image is collected, the image posture can be solved by the computing device loaded on the data collection vehicle. The solution result includes the initial posture of the image and the relative posture within the trip. The initial posture of the image and the relative posture within the trip obtained by the existing image posture solution scheme mastered by those skilled in the art can be used in the scheme provided by the present disclosure. The present disclosure has no restrictions on the posture solution scheme.

[0057] In some embodiments, the initial posture can be a posture relative to the world coordinate system calculated by a visual odometer based on multiple consecutive frames of images and the acquisition position corresponding to each frame in the multiple consecutive frames of images. Taking into account the influence of factors such as inaccurate acquisition position positioning when acquiring images, the initial posture has a certain error.

[0058] The relative posture within a pass can be the relative posture between multiple consecutive frames of images, or the relative posture of each frame in the multiple consecutive frames of images relative to the first frame of the image in the data collected in the pass, and the relative posture within a pass can be a posture with a smaller error. For example, the relative posture within a pass of a certain frame of a continuous multiple frame of images can be understood as the relative posture of the frame of the image relative to the first frame of the continuous multiple frames of images, or the relative posture of the frame of the image relative to the previous frame of the frame of the continuous multiple frames of images. The relative posture within a pass can be calculated based on the information output by the inertial measurement device, the positioning device, etc. that is fixedly connected to the image sensor device that collects the image, and the accuracy of the relative posture within a pass is high.

[0059] In some embodiments, the collected data may also include image feature points detected on each frame of the continuous multiple frames of images, and the two-dimensional image coordinates of the image feature points on the frame of image are known. It should be understood that the posture involved in the embodiments of the present disclosure includes the position and posture in the world coordinate system.

[0060] In this embodiment, each frame of image may include multiple image feature points. In this embodiment, each image feature point included in the collected data may include the two-dimensional pixel coordinates of the image feature point. and m-dimensional feature descriptors n represents the number of image feature points in the frame image. It is understandable that the number of image feature points on each frame image may be different. In some embodiments, the image feature point may refer to a point with more information in the image, such as a point where the image grayscale value changes dramatically or a point with a larger curvature on the edge of the image; the feature descriptor may be a feature description vector obtained by extracting features from the image, which may be understood as a set of values ​​that record the characteristics of the image feature point and may be used to match the image feature points in different images.

[0061] In some embodiments, after acquiring two rounds of collected data for the same road, one round of data can be arbitrarily selected from the two rounds of collected data as reference data, and the other round of collected data can be used as data to be associated, and the data to be associated is associated with the reference data. Associating the data to be associated with the reference data can be understood as matching the image feature points that appear in multiple consecutive frames of images in the data to be associated with the image feature points that appear in multiple consecutive frames of images in the reference data, to obtain a matching relationship between the image feature points in the data to be associated and the reference data.

[0062] In some embodiments, three-dimensional reconstruction technology can be used to reconstruct the three-dimensional spatial coordinates corresponding to the image feature points based on the initial pose of each frame image in the reference data and the image feature points in the frame image to obtain three-dimensional point cloud data in each frame image in the reference data.

[0063] In some embodiments, a small segment of data, i.e., a plurality of consecutive first associated images, may be selected from the data to be associated to be associated with the reference data. In some embodiments, a plurality of consecutive images may be selected from the first frame of the data to be associated as the first associated image. In other embodiments, a plurality of consecutive images may be selected from a non-first frame of the data to be associated, i.e., from a certain frame in the middle, as the first associated image.

[0064] In this embodiment, multiple frames of first reference images having a suspected co-viewing relationship with the continuous multiple frames of first associated images can be selected from the reference data according to the initial positions of the continuous multiple frames of first associated images. In some embodiments, whether there is a co-viewing relationship between two frames of images can be determined by the acquisition position corresponding to the image and / or the similarity between the images. For a certain frame of target image in the first associated image selected from the data to be associated, if there is one or more reference images in the reference data, which are close to the acquisition position of the frame of target image, such as the position difference is less than the set distance threshold, and / or the image similarity between the reference image and the target image is greater than or equal to the set similarity threshold, then the reference image can be considered as an image having a co-viewing relationship with the target image. It can be understood that for each frame of the first associated image in the data to be associated, more than one reference image can be found from the reference data. Of course, it can also be understood that when there are multiple reference images in the reference data that have a co-viewing relationship with the first associated image in the data to be associated, a frame of image with the closest position and / or the highest similarity can be selected from the multiple reference images as the first reference image having a co-viewing relationship with the first associated image.

[0065] After selecting the first associated image in the data to be associated and the first reference image in the reference data that has a suspected co-viewing relationship with the first associated image, the image feature points that match between the first associated image and the first reference image, as well as the target pose of the first associated image can be determined based on the initial pose of the first reference image, the image feature points appearing on the first reference image and their corresponding three-dimensional point cloud data, the initial pose of the first associated image, the image feature points of the first associated image, the camera intrinsic parameters for collecting the first associated image, and the intra-pass relative pose of the first associated image.

[0066] In some embodiments, the first associated image includes a continuous plurality of frames, and the relative posture within the first associated image can be understood as the relative posture of each frame in the continuous plurality of frames to the first frame in the continuous plurality of frames, or the relative posture of each frame relative to any frame in the continuous plurality of frames, for example, the relative posture of each frame relative to the previous frame.

[0067] Since the initial poses of each frame image in any two rounds of collected data are different and the initial pose is the pose calculated based on the image and the corresponding positioning data, the accuracy is not high. If for each frame image in the data to be associated, the pose is optimized by matching with the image in the reference data, and the optimized pose is used for association, it will consume a lot of computing resources and the cost is high. Therefore, the embodiment of the present disclosure first selects part of the images, that is, a plurality of consecutive frames of first associated images, from the data to be associated, and associates them with the reference data, that is, determines the image feature points that match between the first associated image and the first reference image in the reference data. Afterwards, the rough pose of the remaining image in the data to be associated, that is, the second associated image, is inferred based on the target pose of the first associated image, and the inferred rough pose is used to associate with the reference data, that is, the image feature points that match between the second associated image and the second reference image in the reference data are determined, and finally all associated images in the associated data are associated with the reference image in the reference data. It can be understood that in this embodiment, the associated image and the reference image are associated, which can be understood as determining the image feature points that match between each frame of the associated image in the associated data and the corresponding reference image in the reference data, and the associated image and the reference image are associated through the matching image feature points.

[0068] In some embodiments, the second associated image may be an image adjacent to the first associated image in the to-be-associated data, or an image not adjacent to the first associated image. In this embodiment, the number of the second associated images may be one or more. When the number of the second associated images is multiple, the following two possible situations may be included:

[0069] (1) The second associated image is an image in the to-be-associated data that is adjacent to the first associated image. In this case, the first frame of the multiple frames included in the second associated image is adjacent to the last frame of the first associated image, and the images other than the first frame in the multiple frames may be adjacent or non-adjacent; or the last frame of the multiple frames included in the second associated image is adjacent to the first frame of the first associated image, and the images other than the last frame in the multiple frames may be adjacent or non-adjacent.

[0070] (2) The second associated image is an image in the to-be-associated data that is not adjacent to the first associated image, that is, the multiple frames of images included in the second associated image are not adjacent to the last frame of the first associated image, and the multiple frames of images may be adjacent or non-adjacent.

[0071] For the second associated image in the data to be associated, the rough pose of the second associated image can be calculated based on the target pose of the first associated image and the intra-pass relative pose of the second associated image, and then the image feature points that match between the second associated image and the second reference image can be determined based on the image feature points of the second associated image, the rough pose, the camera internal parameters for collecting the second associated image, and the initial pose of the second reference image in the reference data that has a co-viewing relationship with the second associated image, the image feature points of the second reference image and its corresponding three-dimensional point cloud data.

[0072] It can be understood that since the target pose of the first associated image has been determined, the target pose is a relatively accurate pose. Based on the target pose of the first associated image and the intra-pass relative pose of the second associated image, the rough pose of the second associated image can be inferred. Based on the rough pose, the second associated image can be matched with the image in the reference data to find the second reference image that has a common visual relationship with the second associated image, and the matching image feature points in the second associated image and the second reference image can be determined.

[0073] When there are other associated images that are not associated with the reference data in the data to be associated, the second associated image whose matching image feature points are determined in the previous round can be used as a new first associated image, and the target pose of the new first associated image is determined using the matching image feature points and the three-dimensional point cloud data corresponding to the image feature points in the second reference image that matches the image feature points. Then, a part of the other associated images that are not associated with the reference data in the data to be associated is taken out or all other remaining associated images are taken out as new second associated images, and the above steps are repeated to determine the image feature points that match between the new second associated image and the new second reference image based on the target pose of the new first associated image, the relative pose within the trip of the new second associated image, the image feature points of the new second associated image, the camera internal parameters for collecting the new second associated image, the initial pose of the new second reference image in the reference data, the image feature points and their corresponding three-dimensional point cloud data, etc. In this way, the image feature points of the second associated images other than the first associated image in the data to be associated and the second reference image in the reference data that match each other can be obtained, and finally the image feature points that match each frame of associated image in the data to be associated and the corresponding reference image in the reference data can be determined, thereby establishing an association relationship between the images in the data to be associated and the reference data.

[0074] In some embodiments, the intra-pass relative posture of the second associated image can be understood as the relative posture of the second associated image relative to the first frame image in the data to be associated, or the relative posture of the second associated image relative to other images in the data to be associated, for example, it can be the relative posture relative to the previous frame image of the second associated image.

[0075] The disclosed embodiment provides a data association method, which can select one of the collected data as reference data and the other collected data as data to be associated after acquiring two collected data of the same road. Using three-dimensional reconstruction technology, three-dimensional point cloud data corresponding to image feature points in the reference data is obtained. In order to align the pose of the data to be associated with the reference data, multiple frames of images can be selected from the data to be associated as first associated images, and multiple frames of first reference images having a co-viewing relationship with the first associated image can be selected from the reference data. Then, based on the initial pose of the first reference image, the image feature points on the first reference image and the three-dimensional point cloud data corresponding to the image feature points, the initial pose of the first associated image, the image feature points, the first relative pose of the first associated image, and the camera internal parameters of the first associated image, the image feature points that match between the first associated image and the first reference image and the target pose of the first associated image are determined after aligning the first associated image with the corresponding first reference image. Afterwards, the rough pose of the second associated image can be determined based on the target pose of the first associated image and the relative pose within the pass of the second associated image other than the first associated image in the data to be associated. The image feature points that match between the second associated image and the second reference image can be determined based on the image feature points, rough pose, camera internal parameters for collecting the second associated image, the initial pose of the second reference image in the reference data that has a co-viewing relationship with the second associated image, image feature points, and three-dimensional point cloud data corresponding to the image feature points. All associated images in the data to be associated are processed in the same manner as above, so that the reference data and the associated data can be preliminarily associated and aligned. The above method provided by the embodiment of the present disclosure can efficiently and simply realize the association of image feature points of images in two passes of collected data by first associating and aligning some images in the data to be associated with the reference data, and then associating and aligning other images in the data to be associated with the reference data based on the aligned partial images.

[0076] In an optional implementation of this embodiment, step S105, i.e., the step of determining the image feature points that match between the first reference image and the first associated image, and the target pose of the first associated image based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, the intra-pass relative pose of the first associated image and the camera intrinsic parameters for acquiring the first associated image, may further include the following steps:

[0077] Based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, and relative pose within the first associated image, and the intrinsic parameters of the camera used to collect the first associated image, the pose of the first associated image is rigidly aligned to the first reference image to obtain a coarse pose of the first associated image;

[0078] Based on the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the coarse pose of the first associated image, the image feature points, and the camera intrinsic parameters for acquiring the first associated image, the image feature points that match between the first reference image and the first associated image and the target pose of the first associated image are determined.

[0079] In this optional implementation, the initial pose of the first associated image can be used to retrieve a first reference image that has a co-viewing relationship with the first associated image from the reference data. Thereafter, the first associated image and the first reference image can be rigidly aligned in pose to obtain a rough pose of the first associated image after rigid alignment. The rigid alignment of posture can be understood as matching the image feature points of the first associated image with the image feature points of the first reference image through the initial posture, image feature points of the first associated image, the initial posture, image feature points of the first reference image and the three-dimensional point cloud data corresponding to the image feature points. For example, the image feature points that are preliminarily matched with the first associated image and the first reference image can be determined by calculating the similarity between the feature descriptors of the image feature points on the first associated image and the feature descriptors of the image feature points on the first reference image. Then, based on the three-dimensional point cloud data corresponding to the preliminarily matched image feature points, algorithms such as RANSAC are used to filter out the external points with large matching errors (that is, the three-dimensional points with large matching errors), and retain the internal points with small matching errors (that is, the three-dimensional points with small matching errors). The image feature points corresponding to the retained three-dimensional points, the three-dimensional point cloud data corresponding to the image feature points and the camera internal parameters for collecting the first associated image can be used to determine the rough posture of the first associated image. The rough posture can be understood as the posture obtained after the first associated image and the first reference image are rigidly aligned.

[0080] After determining the rough pose, the rough pose can be used to re-determine the image feature points that match between the first associated image and the first reference image, and then determine a more accurate target pose of the first associated image and the image feature points that match between the first associated image and the first reference image based on the re-matched image feature points and the three-dimensional point cloud data corresponding to the image feature points, the camera intrinsic parameters for collecting the first associated image, and the intra-pass relative pose of the first associated image.

[0081] In an optional implementation of this embodiment, based on the initial pose of the first reference image, the image feature points and the three-dimensional point cloud data corresponding to the image feature points, the initial pose of the first associated image, the image feature points, the intra-pass relative pose, and the camera intrinsic parameters of the first associated image, the pose of the first associated image is rigidly aligned to the first reference image, and the step of obtaining the coarse pose of the first associated image may further include the following steps:

[0082] Based on the initial position and posture of the first reference image, the image feature points and the three-dimensional point cloud data corresponding to the image feature points, and the initial position and posture of the first associated image and the image feature points, obtain image feature points having a rough matching relationship between the first associated image and the first reference image;

[0083] Based on the image feature points with a rough matching relationship, the three-dimensional point cloud data of the image feature points, the camera intrinsic parameters for collecting the first associated image, and the intra-pass relative pose of the first associated image, the rough pose of the first associated image is determined after the pose of the first associated image is rigidly aligned to the first reference image.

[0084] In this optional implementation, the image feature points in the first associated image and the first reference image can be matched by using the initial position of the first associated image, the initial position of the first reference image, the image feature points of the first reference image, the image feature points of the second associated image, and the three-dimensional point cloud data corresponding to the image feature points of the first reference image to obtain image feature points with a rough matching relationship between the first associated image and the first reference image. In this process, the image feature points of the first associated image and the first reference image with which it has a co-viewing relationship can be matched first to obtain the image feature points with preliminary matching. Then, the RANSAC algorithm can be used to filter out the external points (i.e., the point cloud data with a large matching error) in the three-dimensional point cloud data corresponding to the image feature points with a preliminary matching relationship, and retain the internal points (i.e., the point cloud data with a small matching error). The image feature points corresponding to the retained internal points are the image feature points with a rough matching relationship between the first associated image and the first reference image that are finally determined.

[0085] Based on the image feature points with a rough matching relationship and their corresponding three-dimensional point cloud data, the relative pose of the first associated image within the pass and the camera internal parameters for collecting the first associated image, the rough pose of the first associated image after the pose of the first associated image is rigidly aligned to the first reference image can be solved. The three-dimensional point cloud data of the image feature points with a rough matching relationship can be determined based on the three-dimensional point cloud data of the image feature points in the first reference image.

[0086] In an optional implementation of this embodiment, the step of determining the image feature points that match between the first reference image and the first associated image and the target pose of the first associated image based on the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the coarse pose of the first associated image, the image feature points, and the camera intrinsic parameters for acquiring the first associated image may further include the following steps:

[0087] Based on the image feature points in the first associated image, the image feature points of the first reference image and the three-dimensional point cloud data corresponding to the image feature points, the coarse pose of the first associated image and the camera intrinsic parameters for acquiring the first associated image, the three-dimensional point cloud data corresponding to the image feature points in the first reference image are projected onto the first associated image, image feature points whose reprojection errors are greater than a set value are filtered out, and image feature points whose matching relationship is accurate between the first reference image and the first associated image are determined;

[0088] The target posture of the first associated image is determined based on the image feature points with accurate matching relationship and the three-dimensional point cloud data corresponding to the image feature points of the first reference image.

[0089] In this optional implementation, after the coarse pose of the first associated image is determined, the three-dimensional point cloud data of the image feature points of the first reference image can be projected onto the first associated image based on the coarse pose, the image feature points of the first associated image, the image feature points in the first reference image and their corresponding three-dimensional point cloud data, and the camera intrinsic parameters for acquiring the first associated image. If the projection error between the projected point obtained after the projection and the corresponding image feature point on the first associated image is greater than a set value, the image feature point is filtered out, and the image feature points whose projection error is less than or equal to the set value are retained, so that the retained image feature points are the image feature points with an accurate matching relationship between the first reference image and the first associated image.

[0090] After obtaining the image feature points with a relatively accurate matching relationship, the three-dimensional point cloud data of the image feature points with a relatively accurate matching relationship can be determined based on the three-dimensional point cloud data corresponding to the image feature points in the first reference image, and then the target posture of the first associated image can be solved based on the image feature points with a relatively accurate matching relationship and their corresponding three-dimensional point cloud data.

[0091] In an optional implementation of this embodiment, based on the image feature points with a coarse matching relationship, the three-dimensional point cloud data of the image feature points, the camera intrinsic parameters for collecting the first associated image, and the intra-pass relative pose of the first associated image, after determining that the pose of the first associated image is rigidly aligned to the first reference image, the step of determining the coarse pose of the first associated image may further include the following steps:

[0092] Determining the candidate pose of each first associated image after the pose of each first associated image is rigidly aligned with the corresponding first reference image based on the image feature points having a coarse matching relationship, the three-dimensional point cloud data corresponding to the image feature points, and the camera intrinsic parameters for acquiring the first associated image;

[0093] Determine a first-frame optimized pose of a first frame of the first associated images based on the candidate poses of each of the first associated images and the intra-pass relative poses of the first associated images;

[0094] Based on the first frame optimized pose and the intra-pass relative pose of the first associated image, a coarse pose of the first associated image is determined.

[0095] In this optional implementation, the candidate pose of the first associated image is calculated based on the image feature points with a rough matching relationship in each frame of the first associated image, the three-dimensional point cloud data corresponding to the image feature points, and the camera internal parameters for collecting the first associated image. In some embodiments, the corresponding candidate pose can be calculated for each frame of the first associated image. The candidate pose can be understood as the pose of the first associated image relative to the world coordinate system after the first associated image is rigidly aligned with the pose of the first reference image.

[0096] In some embodiments, a PnP (Perspective-n-Point) algorithm can be used to obtain a candidate pose of the first associated image based on image feature points with a rough matching relationship in the first associated image, three-dimensional point cloud data corresponding to the image feature points, and camera intrinsic parameters collected for the first associated image.

[0097] Each frame of the first associated image corresponds to a candidate pose, and based on the intra-pass relative pose of each candidate pose and the first associated image, the first frame optimized pose corresponding to the first frame image in the first associated image can be determined.

[0098] In some embodiments, after obtaining the optimized pose of the first frame, the rough pose of the first associated image can be further optimized based on the optimized pose of the first frame and the relative pose within the pass of the first associated image. It should be understood that since the optimized pose of the first frame is a pose with higher accuracy than the initial pose of the first frame image in the first associated image after optimization, the rough pose of the first associated image determined based on the optimized pose of the first frame and the relative pose within the pass of the first associated image has a higher accuracy rate than the initial pose of the first associated image and the candidate poses, that is, the rough pose of the first associated image can be understood as a pose after optimizing the initial pose of the first associated image.

[0099] In an optional implementation of this embodiment, the step of determining the first frame optimized pose of the first frame image in the first associated images based on the candidate pose of each of the first associated images and the intra-pass relative pose of the first associated images may further include the following steps:

[0100] Determine a plurality of first-frame candidate poses of a first frame image in the first associated images based on each of the candidate poses of each of the first associated images and the intra-pass relative poses of the first associated images;

[0101] Obtaining multiple reference poses of each of the first associated images by using multiple first frame candidate poses and the intra-pass relative poses of the first associated images;

[0102] Based on the deviation between the candidate pose of each of the first associated images and the corresponding reference pose, the first frame optimized pose of the first frame image is screened out from the plurality of first frame candidate poses.

[0103] In this optional implementation, since the initial pose of the first associated image may not be accurate in itself, the calculated image feature points with a rough matching relationship may still contain errors, which may cause noise in the solved candidate pose. Therefore, the accuracy of the candidate pose can be verified by using the relatively accurate intra-pass relative pose of the data to be associated, that is, the intra-pass relative pose of the first associated image. Therefore, based on the candidate pose and the intra-pass relative pose corresponding to multiple frames of images in the first associated image, a relatively accurate first-frame optimized pose of the first frame image in the first associated image can be determined. That is to say, the relative posture of each frame in the first associated image, no matter whether it is initially relative to the relative posture of the first frame image in the data to be associated or relative to other images in the data to be associated, can be converted into the relative posture of each frame in the first associated image relative to the first frame image in the first associated image. Then, based on the candidate postures of multiple frames of first associated images and the relative postures relative to the first frame image, the solved posture of the first frame image can be obtained. Multiple frames of first associated images can calculate multiple first-frame candidate postures of the first frame image, and the one with the highest accuracy can be selected from the multiple first-frame candidate postures as the first-frame optimized posture of the first frame image.

[0104] In some embodiments, as described above, the intra-pass relative pose between each first associated image and the first frame image in the first associated image can be obtained based on the intra-pass relative pose of the first associated image, and thus based on the intra-pass relative pose between the first frame image and the multiple first-frame candidate poses of the first frame image, multiple reference poses of each first associated image can be obtained, and based on the deviations between the multiple reference poses and the candidate poses of the first associated image, the optimized first-frame pose is selected from the multiple first-frame candidate poses.

[0105] In some embodiments, multiple first frame candidate poses of the first frame image in the first associated image can be calculated based on the following formula:

[0106]

[0107] in, is the candidate pose of the j-th frame image in the first associated image, is the relative position of the first frame image of the first associated image relative to the jth frame image in the first associated image.

[0108] In other embodiments, after obtaining multiple first-frame candidate poses of the first frame image, multiple reference poses of the first associated image can be calculated based on the multiple first-frame candidate poses and the intra-pass relative pose of the first associated image. Then, since there are deviations between the multiple reference poses of the first associated image and the candidate poses of the first associated image, a deviation threshold can be set in advance to determine the target number whose deviations between the multiple reference poses corresponding to each first-frame candidate pose and the corresponding candidate poses are less than or equal to the deviation threshold, that is, each first-frame candidate pose corresponds to a target number, and the first-frame candidate pose corresponding to the largest target number is used as the first-frame optimized pose of the first frame image.

[0109] In some embodiments, the deviation threshold can be represented by ε, and the first frame optimized pose of the first frame image can be obtained by the following formula: w T 0 :

[0110]

[0111] Among them, dis(a,b) represents the Euclidean position distance between the two positions a and b. is a sign function. When the Euclidean position distance between the above two postures is less than the deviation threshold ε, the value of a is 1, and when it is not satisfied, the value of a is 0. is the candidate pose of the jth image in the first associated image, is the intra-pass phase position of the j-th frame image of the first associated image relative to the first frame image in the first associated image, and n is the number of images in the first associated image.

[0112] In an optional implementation of this embodiment, based on the image feature points in the first associated image, the image feature points of the first reference image and the three-dimensional point cloud data corresponding to the image feature points, the coarse pose of the first associated image and the camera intrinsic parameters for acquiring the first associated image, the three-dimensional point cloud data corresponding to the image feature points in the first reference image are projected onto the first associated image, image feature points whose reprojection errors are greater than a set value are filtered out, and the step of determining the image feature points with accurate matching relationships between the first reference image and the first associated image may further include the following steps:

[0113] Re-determining image feature points having a rough matching relationship between the first associated image and the first reference image based on image feature points in the first associated image, image feature points in the first reference image, three-dimensional point cloud data corresponding to the image feature points, and the rough pose;

[0114] Based on the re-determined image feature points with a rough matching relationship, the three-dimensional point cloud data corresponding to the image feature points, the camera intrinsic parameters for collecting the first associated image, and the rough pose of the first associated image, projecting the three-dimensional point cloud data corresponding to the re-determined image feature points with a rough matching relationship onto the first associated image;

[0115] Determine a reprojection error between a projection point on the first associated image and the image feature point having a rough matching relationship re-determined in the first associated image;

[0116] The image feature points whose reprojection errors are greater than the set value are filtered out from the re-determined image feature points with a rough matching relationship, so as to obtain image feature points with an accurate matching relationship between the first reference image and the first associated image.

[0117] In this optional implementation, after optimizing and obtaining a more accurate rough pose of the first associated image, the image feature points of the first associated image and the image feature points of the first reference image can be re-matched based on the rough pose. For example, the image feature points with a preliminary matching relationship between the first associated image and the first reference image can be re-determined by calculating the similarity between the feature descriptors of the image feature points on the first associated image and the feature descriptors of the image feature points on the first reference image, and then the three-dimensional point cloud data of the image feature points with a preliminary matching relationship can be determined based on the three-dimensional point cloud data corresponding to the image feature points on the first reference image. After that, the external points (i.e., the point cloud data with a large matching error) in the three-dimensional point cloud data can be filtered out by the RANSAC algorithm, and then the image feature points corresponding to the retained internal points (i.e., the remaining three-dimensional point cloud data with a small matching error) and the image feature points on the first associated image with a preliminary matching relationship can be determined based on the correspondence between the image feature points with a preliminary matching relationship and the three-dimensional point cloud data of the first reference image, so that the image feature points with a rough matching relationship between the first associated image and the first reference image can be obtained.

[0118] In some embodiments, after the image feature points with a rough matching relationship between the first associated image and the first reference image are re-determined based on the rough pose, the three-dimensional point cloud data of the re-determined image feature points with a rough matching relationship can be determined based on the three-dimensional point cloud data corresponding to all the image feature points in the first reference image, and then the three-dimensional point cloud data corresponding to the image feature points with a rough matching relationship can be projected onto the first associated image to obtain a reprojection point based on the three-dimensional point cloud data of the image feature points with a rough matching relationship, the rough pose of the first associated image, and the camera intrinsic parameters for collecting the first associated image, and the reprojection error between the projection point and the corresponding image feature point on the first associated image is calculated, and the image feature points whose corresponding reprojection errors in the three-dimensional point cloud data are greater than a set value are filtered out from the image feature points with a rough matching relationship of the first associated image, and the image feature points with a rough matching relationship that are finally retained are image feature points with a relatively accurate matching relationship. The projection error can be represented by δ, for example, δ can be taken as 30, that is, when the projection error is greater than 30 pixels, the image feature point can be filtered out.

[0119] In an optional implementation of this embodiment, the step of determining the target pose of the first associated image based on the image feature points with accurate matching relationship and the three-dimensional point cloud data corresponding to the image feature points of the first reference image may further include the following steps:

[0120] Based on the three-dimensional point cloud data corresponding to the image feature points with accurate matching relationship and the image feature points of the first reference image, the pre-optimization pose of the first associated image is calculated;

[0121] The pre-optimization pose of the first associated image and the intra-pass relative pose of the first associated image are used as constraints, and the pre-optimization pose of the first associated image is optimized by constructing a pose optimization graph to obtain a target pose of the first associated image.

[0122] In this optional implementation, after obtaining the image feature points with a relatively accurate matching relationship between the first associated image and the first reference image, the three-dimensional point cloud data corresponding to the image feature points with an accurate matching relationship in the first associated image can be determined based on the association relationship between the image feature points on the first reference image and the three-dimensional point cloud data. In other words, the three-dimensional point cloud data corresponding to the image feature points on the first associated image are the three-dimensional point cloud data corresponding to the image feature points of the first reference image that match the image feature points of the first associated image.

[0123] In some embodiments, after obtaining the three-dimensional point cloud data of the image feature points with an accurate matching relationship in the first associated image, the pose calculation algorithm can be used to calculate the pose of the first associated image before optimization based on the three-dimensional point cloud data and the corresponding image feature points on the first associated image. In this embodiment, the pose calculation algorithm can be a PnP (Perspective-n-Point) algorithm. It should be understood that the pose before optimization is the result of further optimization based on the rough pose, so the pose before optimization is more accurate than the rough pose.

[0124] Considering that the pose before optimization is obtained by matching the image feature points and filtering out the points with large reprojection errors, there is a certain error. Therefore, the pose before optimization can be further optimized using the relatively accurate relative pose within the first associated image. In some embodiments, the optimization can be regarded as a pose graph optimization problem, so as to obtain the target pose of the first associated image based on the pose before optimization of the first associated image and the relative pose within the first associated image.

[0125] In this embodiment, the pre-optimization posture is further tested and optimized based on the relatively accurate intra-pass relative posture of the first associated image, so that the accuracy of the target posture of the first associated image is further improved.

[0126] In an optional implementation of this embodiment, the step of taking the pre-optimization pose of the first associated image and the intra-pass relative pose of the first associated image as constraints, optimizing the pre-optimization pose of the first associated image by constructing a pose optimization graph, and obtaining a target pose of the first associated image may further include the following steps:

[0127] The pre-optimization posture and the intra-pass relative posture are used as constraints, and the target posture of the first associated image is optimized and solved by constructing a posture optimization graph.

[0128] In this optional implementation, after obtaining the pre-optimization pose of the first associated image and the intra-pass relative pose of the first associated image, a pose graph may be constructed to further optimize and solve the target pose of the first associated image. The specific formula is as follows:

[0129]

[0130] in, is the relative position of the i+1th frame image relative to the ith frame image in the first associated image, w T i newprior is the pre-optimization pose of the i-th frame image in the first associated image, which is the pose relative to the world coordinate system. w T i is the pose of the i-th frame in the first associated image relative to the world coordinate system, that is, the target pose to be optimized. is the position of the world coordinate system relative to the i+1th frame in the first associated image, is the position of the world coordinate system relative to the i-th frame image in the first associated image.

[0131] It should be noted that the target pose of the first associated image is obtained after optimization based on the pose before optimization of the first associated image, so the target pose of the first associated image is more accurate than the pose before optimization of the first associated image.

[0132] In an optional implementation of this embodiment, step S107, i.e., the step of determining the image feature points matching between the second associated image and the second reference image based on the image feature points, the coarse pose, the camera intrinsic parameters for acquiring the second associated image, and the initial pose, the image feature points, and the three-dimensional point cloud data corresponding to the image feature points of the second reference image having a co-viewing relationship with the second associated image in the reference data, may further include the following steps:

[0133] Determine image feature points having a rough matching relationship between the second associated image and the second reference image by using the image feature points and the rough pose of the second associated image as well as the initial pose, image feature points of the second reference image and the three-dimensional point cloud data corresponding to the image feature points;

[0134] Based on the image feature points with a rough matching relationship, the three-dimensional point cloud data corresponding to the image feature points of the second reference image, the rough pose of the second associated image and the camera intrinsic parameters for acquiring the second associated image, the three-dimensional point cloud data corresponding to the image feature points with a rough matching relationship are projected onto the second associated image, and from the image feature points with a rough matching relationship, image feature points whose reprojection errors are greater than a set value are filtered out to obtain image feature points with an accurate matching relationship between the second reference image and the second associated image.

[0135] In this optional implementation, after the target pose of the first associated image is obtained, the rough pose of the second associated image can be obtained based on the target pose of the first associated image and the intra-pass relative pose of the second associated image. It should be understood that since the target pose of the first associated image is an optimized pose with high accuracy, and the intra-pass relative poses between the frames of images in the data to be associated are also relatively accurate, the obtained rough pose of the second associated image is more accurate than the initial pose of the second associated image.

[0136] In this optional implementation, after obtaining the rough pose of the second associated image, the rough pose of the second associated image can be used to retrieve the second reference image that has a co-viewing relationship with the second associated image from the reference data. The image feature points of the second associated image can be matched with the image feature points of the second reference image that has a co-viewing relationship through the rough pose, image feature points of the second associated image, the initial pose, image feature points of the second reference image, and the three-dimensional point cloud data corresponding to the image feature points. For example, the image feature points on the second reference image that are preliminarily matched with the image feature points of the second associated image can be determined by calculating the similarity between the feature descriptors of the image feature points on the second associated image and the feature descriptors of the image feature points on the second reference image. Since this feature matching method may have errors, in some embodiments, the RANSAC algorithm can also be used to filter out the image feature points that are matched incorrectly. First, based on the three-dimensional point cloud data corresponding to the preliminary matched image feature points and the image feature points of the second reference image, the three-dimensional point cloud data of the preliminary matched image feature points is determined, and then the three-dimensional point cloud data of the preliminary matched image feature points is processed using the RANSAC algorithm to filter out the external points (that is, the three-dimensional points with larger errors) and retain the internal points (that is, the three-dimensional points with smaller errors). The image feature points corresponding to the retained internal points can be understood as image feature points having a rough matching relationship between the second associated image and the second reference image.

[0137] After obtaining the rough pose of the second associated image and the image feature points with a rough matching relationship between the second associated image and the second reference image, the three-dimensional point cloud data of the image feature points with a rough matching relationship can be projected onto the second associated image to obtain reprojection points based on the image feature points with a rough matching relationship, the rough pose of the second associated image, and the camera internal parameters of the second associated image, and the projection error between the reprojection point and the image feature points with a rough matching relationship in the second associated image is calculated, and the image feature points with a projection error greater than a set value are filtered out, and the remaining image feature points in the image feature points with a rough matching relationship are determined as image feature points with an accurate matching relationship between the second associated image and the second reference image. The projection error can be represented by δ, for example, δ can be taken as 30, that is, when the projection error is greater than 30 pixels, the image feature point can be filtered out.

[0138] If the to-be-associated data also includes an unprocessed associated image, the target posture of the second associated image is determined based on the image feature points matching the second associated image and the three-dimensional point cloud data corresponding to the image feature points.

[0139] In this embodiment, the pre-optimization pose of the second associated image can be calculated based on the image feature points with accurate matching relationships, the three-dimensional point cloud data corresponding to the image feature points, and the camera intrinsic parameters for collecting the second associated image.

[0140] Considering that the pose before optimization of the second associated image is obtained by matching the image feature points and filtering out the points with large reprojection errors, there is a certain error. Therefore, the pose before optimization can be further optimized using the relatively accurate relative pose within the second associated image. In some embodiments, the optimization can be regarded as a pose graph optimization problem, so as to optimize the target pose of the second associated image based on the pose before optimization of the second associated image and the relative pose within the second associated image.

[0141] In this embodiment, the pre-optimization posture is further verified and optimized based on the relatively accurate intra-pass relative posture of the second associated image, so that the accuracy of the target posture of the second associated image is further improved.

[0142] In some embodiments, a pose graph may be constructed to further optimize and solve the pre-optimization pose of the second associated image. The specific formula is as follows:

[0143]

[0144] in, is the relative position of the i+1th frame relative to the ith frame in the second associated image, w Ti newprior is the pre-optimization pose of the i-th frame image in the second associated image, which is the pose relative to the world coordinate system. w T i is the pose of the i-th frame in the second associated image relative to the world coordinate system, that is, the target pose. is the position of the world coordinate system in the second associated image relative to the i+1th frame, is the pose of the world coordinate system in the second associated image relative to the i-th frame.

[0145] It should be noted that the target pose of the second associated image is obtained after optimization based on the pose of the second associated image before optimization, so the target pose of the second associated image is more accurate than the pose of the second associated image before optimization.

[0146] Figure 2 FIG. 2 is a schematic diagram showing a complete process implementation of a data association method according to an embodiment of the present disclosure. Figure 2 As shown, the method can generally include the following four processes:

[0147] (1) Data collection stage

[0148] At least one vehicle traveling on the same road is obtained, and multiple trips of data collected on the road during the driving process are obtained. Each trip of data can be transmitted back by the vehicle to the server. Each trip of data transmitted back by the vehicle may include but is not limited to multiple frames of images continuously collected within a period of time during the driving process on the road and sensor information corresponding to the camera when collecting the multiple frames of images. The sensor information may include but is not limited to positioning information output by a positioning sensor and an inertial measurement unit fixedly connected to the camera, inertial measurement information, etc. The vehicle can transmit the continuous multiple frames of images and sensor information in each trip of data to the server, and the server can use the sensor information to generate the initial posture of the continuous multiple frames of images in each trip of data and the relative posture within the trip. The initial posture can be the posture of each frame of the continuous multiple frames of images relative to the world coordinate system or other unified coordinate systems, and the relative posture within the trip can be the relative posture of each frame of the image relative to the first frame of the image or other images in the data collected in the trip. The server can also extract feature points for the continuous multiple frames of images to obtain information such as image feature points in each frame of image and feature descriptors of image feature points.

[0149] (2) Data preprocessing stage of reference data

[0150] The collected data that the server can obtain for each trip of data sent back by the vehicle may include but is not limited to continuous multiple frames of images on the road, the initial position of the continuous multiple frames of images, the relative position within the trip, and the image feature points in each frame of the image. After the server obtains multiple trips of collected data for the same road, the multiple trips of collected data can be associated, and then data alignment can be performed based on the association results. In the data association process, one of the multiple trips of collected data can be selected as reference data, and the other collected data can be associated and aligned to the reference data.

[0151] For the reference data, three-dimensional reconstruction technology can be used to reconstruct the three-dimensional spatial coordinates corresponding to the image feature points based on the initial pose of each frame image in the reference data and the image feature points in the frame image, so as to obtain the three-dimensional point cloud data in each frame image in the reference data.

[0152] (3) Initialization of trajectories of some images in the data to be associated

[0153] The server may also use other data in the multi-pass collected data except the reference data as the data to be associated. For the data to be associated, a small portion of images, i.e., a plurality of consecutive first associated images, may be selected from the data to be associated to be associated with the reference data. In some embodiments, a plurality of consecutive images may be selected from the first frame of the data to be associated as the first associated images. In other embodiments, a plurality of consecutive images may be selected from a non-first frame of the data to be associated, i.e., from a certain frame in the middle, as the first associated images.

[0154] Afterwards, multiple frames of first reference images having a suspected common-viewing relationship with the multiple frames of first associated images can be selected from the reference data according to the initial positions of the multiple frames of first associated images.

[0155] After determining the first associated image and the first reference image, the initial pose of the first associated image can be used to determine the three-dimensional point cloud data that matches the image feature points in the first associated image from the three-dimensional point cloud data corresponding to the image feature points on the first reference image; and based on the three-dimensional point cloud data, determine the candidate pose of the first associated image after the first associated image is aligned with the corresponding first reference image. Based on the candidate pose and the intra-pass relative pose of the first associated image, determining the first frame optimized pose of the first frame image in the first associated image ( w T 0 ); and determining the rough pose of the first associated image based on the first frame optimized pose and the intra-pass relative pose of the first associated image ( w T i prior ).

[0156] Afterwards, the rough pose of the first associated image and the three-dimensional point cloud data corresponding to the image feature points on the first reference image can be used to obtain the image feature points having a matching relationship between the first associated image and the corresponding first reference image. Based on the three-dimensional point cloud data corresponding to the image feature points having a matching relationship, the pose before optimization of the first associated image after alignment with the corresponding first reference image is determined ( w T i newprior ), so that the target pose of the first associated image can be optimized based on the pre-optimization pose of the first associated image and the relative pose of the first associated image within the trip.

[0157] (4) Estimation of the pose of the second image in the data to be associated

[0158] After the target pose of the first associated image pose is obtained through optimization, the target pose of the first associated image can be directly used to infer the target pose of the other images in the to-be-associated data except the first associated image, namely the second associated image, and the second associated image can be matched with the second reference image in the reference data. Since the pose obtained by optimizing the target pose of the first associated image after alignment with the reference data is a more accurate pose, and the intra-pass relative pose of the second associated image in the to-be-associated data is also a more accurate relative pose, the accuracy of the result of associating and aligning the target pose of the first associated image, the intra-pass relative pose of the second associated image and the reference data is also high.

[0159] Therefore, in this process, the rough pose of the second associated image can be determined based on the target pose of the first associated image and the intra-pass relative pose of the second associated image; the image in the reference data that has a co-viewing relationship with the second associated image is used as the second reference image, and based on the rough pose of the second associated image, the image feature points of the second associated image and the image feature points of the second reference image and their corresponding three-dimensional point cloud data, the image feature points in the second associated image that have a rough matching relationship with the second reference image are determined. After the three-dimensional point cloud data corresponding to the image feature points with a rough matching relationship are projected onto the second associated image, the image feature points with an accurate matching relationship can be selected from the image feature points with a rough matching relationship based on the projection error between the reprojected point obtained by projection and the candidate feature point, so that the image feature points with an accurate matching relationship can be selected.

[0160] Afterwards, the optimized pose of the second associated image after alignment with the corresponding second reference image can be determined based on the image feature points with accurate matching relationship, the three-dimensional point cloud data corresponding to the image feature points, and the camera intrinsic parameters for collecting the second associated image. Based on the pre-optimization pose of the second associated image and the intra-pass relative pose of the second associated image, the target pose of the second associated image is optimized and solved.

[0161] After the above processing, a matching relationship between the image feature points on the image in the data to be associated and the image feature points on the image in the reference data is established. Based on the matching relationship, the alignment result of the data to be associated and the reference data can be further optimized to obtain a higher alignment result. The embodiment of the present disclosure does not limit the further optimization method, which can be selected based on actual needs.

[0162] This embodiment can be applied to the positioning task of crowdsourcing data. Since accurate positioning and accurate matching of image feature points are dual, accurate positioning means that the matching of image feature points is also accurate, and accurate matching of image feature points means that positioning is accurate. Crowdsourcing data is collected by many ordinary vehicles (which can be called crowdsourcing vehicles) when they are driving on the road. As described above, the collected data may include trajectory data and image data. The trajectory data includes a series of positioning points obtained by the crowdsourcing vehicle through positioning technology during driving, and the image data includes images of the road taken by the crowdsourcing vehicle using the on-board camera during driving. Based on the above trajectory data and image data, the initial pose and relative pose within the trip of the image in the collected data can be solved. Due to the influence of the environment and the positioning differences between different crowdsourcing vehicles, the positioning points in the trajectory data of the crowdsourcing vehicle may not be accurate, so the initial pose obtained by solving may not be accurate, but the relative pose within the trip is accurate. Therefore, the disclosed embodiment utilizes the characteristic of accurate relative posture within a pass to first associate the image feature points of a small part of the images in the two passes of collected data, and then uses the relative posture within the pass to infer the posture of the remaining images, and uses the inferred posture to associate the image feature points of the remaining images, thereby improving the matching accuracy of image feature points, and further improving the fusion positioning accuracy of crowdsourcing data.

[0163] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.

[0164] Figure 3 FIG. 1 is a block diagram showing a data association device according to an embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 3 As shown, the data association device includes:

[0165] The first acquisition module 301 is configured to acquire two trips of collected data on the same road, wherein each trip of collected data includes: multiple frames of images collected on the road, image feature points of each frame of image, and initial posture of each frame of image, and relative posture within the trip;

[0166] The selection module 302 is configured to select one of the two rounds of collected data as reference data and the other round of collected data as data to be associated;

[0167] The first determination module 303 is configured to obtain three-dimensional point cloud data corresponding to the image feature points in the reference data using three-dimensional reconstruction technology;

[0168] The second acquisition module 304 is configured to acquire a plurality of frames of first reference images and first associated images having a common viewing relationship from the reference images included in the reference data and the associated images included in the to-be-associated data;

[0169] The second determination module 305 is configured to determine the image feature points that match between the first reference image and the first associated image, and the target pose of the first associated image based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, the intra-pass relative pose of the first associated image and the camera intrinsic parameters for acquiring the first associated image;

[0170] A third determination module 306 is configured to obtain a rough pose of a second associated image based on the target pose of the first associated image and the intra-pass relative pose of a second associated image other than the first associated image in the to-be-associated data;

[0171] A fourth determination module 307 is configured to determine image feature points that match between the second associated image and the second reference image based on image feature points, a rough pose, an intrinsic parameter of a camera for acquiring the second associated image, an initial pose, an image feature point, and the three-dimensional point cloud data corresponding to the image feature point of the second reference image in the reference data that has a co-viewing relationship with the second associated image;

[0172] A fifth determination module 308 is configured to determine a target pose of the second associated image based on image feature points matching the second associated image and the three-dimensional point cloud data corresponding to the image feature points if the to-be-associated data also includes an unprocessed associated image;

[0173] The sixth determination module 309 is configured to use the second associated image as a new first associated image and the unprocessed associated image as a new second associated image, return the target posture based on the first associated image and the intra-pass relative posture of the second associated image other than the first associated image in the data to be associated, and continue to execute the step of obtaining the rough posture of the second associated image until all the associated images in the data to be associated are processed.

[0174] In this embodiment, the data association method can be executed on a server or on a vehicle-mounted device, and the present disclosure does not impose any restrictions. The data association method is applicable to the scenario of associating image feature points of images in two passes of collected data during the construction of a high-precision map based on vision, and the association results of the image feature points can be used to align the trajectory data corresponding to the image, and the alignment scheme can be a bundle adjustment method based on visual feature points.

[0175] For ease of understanding, the present disclosure briefly introduces data collection. The data collection vehicle can perform multiple trips of data collection on any road in the real world. The collected data usually includes trajectory data and image data, wherein the trajectory data is composed of a series of positioning points formed when the data collection vehicle travels on the road. The positioning technology can adopt multi-source fusion positioning, such as inertial navigation, GNSS, etc. When collecting trajectory data, the image sensing device loaded on the data collection vehicle, such as a camera, can take an image of the road. The image and trajectory data can be matched by the collection time. After the image is collected, the image posture can be solved by the computing device loaded on the data collection vehicle. The solution result includes the initial posture of the image and the relative posture within the trip. The initial posture of the image and the relative posture within the trip obtained by the existing image posture solution scheme mastered by those skilled in the art can be used in the scheme provided by the present disclosure. The present disclosure has no restrictions on the posture solution scheme.

[0176] In some embodiments, the initial posture can be a posture relative to the world coordinate system calculated by a visual odometer based on multiple consecutive frames of images and the acquisition position corresponding to each frame in the multiple consecutive frames of images. Taking into account the influence of factors such as inaccurate acquisition position positioning when acquiring images, the initial posture has a certain error.

[0177] The relative posture within a pass can be the relative posture between multiple consecutive frames of images, or the relative posture of each frame in the multiple consecutive frames of images relative to the first frame of the image in the data collected in the pass, and the relative posture within a pass can be a posture with a smaller error. For example, the relative posture within a pass of a certain frame of a continuous multiple frame of images can be understood as the relative posture of the frame of the image relative to the first frame of the continuous multiple frames of images, or the relative posture of the frame of the image relative to the previous frame of the frame of the continuous multiple frames of images. The relative posture within a pass can be calculated based on the information output by the inertial measurement device, the positioning device, etc. that is fixedly connected to the image sensor device that collects the image, and the accuracy of the relative posture within a pass is high.

[0178] In some embodiments, the collected data may also include image feature points detected on each frame of the continuous multiple frames of images, and the two-dimensional image coordinates of the image feature points on the frame of image are known. It should be understood that the posture involved in the embodiments of the present disclosure includes the position and posture in the world coordinate system.

[0179] In this embodiment, each frame of image may include multiple image feature points. In this embodiment, each image feature point included in the collected data may include the two-dimensional pixel coordinates of the image feature point. and m-dimensional feature descriptors n represents the number of image feature points in the frame image. It is understandable that the number of image feature points on each frame image may be different. In some embodiments, the image feature point may refer to a point with more information in the image, such as a point where the image grayscale value changes dramatically or a point with a larger curvature on the edge of the image; the feature descriptor may be a feature description vector obtained by extracting features from the image, which may be understood as a set of values ​​that record the characteristics of the image feature point and may be used to match the image feature points in different images.

[0180] In some embodiments, after acquiring two rounds of collected data for the same road, one round of data can be arbitrarily selected from the two rounds of collected data as reference data, and the other round of collected data can be used as data to be associated, and the data to be associated is associated with the reference data. Associating the data to be associated with the reference data can be understood as matching the image feature points that appear in multiple consecutive frames of images in the data to be associated with the image feature points that appear in multiple consecutive frames of images in the reference data, to obtain a matching relationship between the image feature points in the data to be associated and the reference data.

[0181] In some embodiments, three-dimensional reconstruction technology can be used to reconstruct the three-dimensional spatial coordinates corresponding to the image feature points based on the initial pose of each frame image in the reference data and the image feature points in the frame image to obtain three-dimensional point cloud data in each frame image in the reference data.

[0182] In some embodiments, a small segment of data, i.e., a plurality of consecutive first associated images, may be selected from the data to be associated to be associated with the reference data. In some embodiments, a plurality of consecutive images may be selected from the first frame of the data to be associated as the first associated image. In other embodiments, a plurality of consecutive images may be selected from a non-first frame of the data to be associated, i.e., from a certain frame in the middle, as the first associated image.

[0183] In this embodiment, multiple frames of first reference images having a suspected co-viewing relationship with the continuous multiple frames of first associated images can be selected from the reference data according to the initial positions of the continuous multiple frames of first associated images. In some embodiments, whether there is a co-viewing relationship between two frames of images can be determined by the acquisition position corresponding to the image and / or the similarity between the images. For a certain frame of target image in the first associated image selected from the data to be associated, if there is one or more reference images in the reference data, which are close to the acquisition position of the frame of target image, such as the position difference is less than the set distance threshold, and / or the image similarity between the reference image and the target image is greater than or equal to the set similarity threshold, then the reference image can be considered as an image having a co-viewing relationship with the target image. It can be understood that for each frame of the first associated image in the data to be associated, more than one reference image can be found from the reference data. Of course, it can also be understood that when there are multiple reference images in the reference data that have a co-viewing relationship with the first associated image in the data to be associated, a frame of image with the closest position and / or the highest similarity can be selected from the multiple reference images as the first reference image having a co-viewing relationship with the first associated image.

[0184] After selecting the first associated image in the data to be associated and the first reference image in the reference data that has a suspected co-viewing relationship with the first associated image, the image feature points that match between the first associated image and the first reference image, as well as the target pose of the first associated image can be determined based on the initial pose of the first reference image, the image feature points appearing on the first reference image and their corresponding three-dimensional point cloud data, the initial pose of the first associated image, the image feature points of the first associated image, the camera intrinsic parameters for collecting the first associated image, and the intra-pass relative pose of the first associated image.

[0185] In some embodiments, the first associated image includes a continuous plurality of frames, and the relative posture within the first associated image can be understood as the relative posture of each frame in the continuous plurality of frames to the first frame in the continuous plurality of frames, or the relative posture of each frame relative to any frame in the continuous plurality of frames, for example, the relative posture of each frame relative to the previous frame.

[0186] Since the initial poses of each frame image in any two rounds of collected data are different and the initial pose is the pose calculated based on the image and the corresponding positioning data, the accuracy is not high. If for each frame image in the data to be associated, the pose is optimized by matching with the image in the reference data, and the optimized pose is used for association, it will consume a lot of computing resources and the cost is high. Therefore, the embodiment of the present disclosure first selects part of the images, that is, a plurality of consecutive frames of first associated images, from the data to be associated, and associates them with the reference data, that is, determines the image feature points that match between the first associated image and the first reference image in the reference data. Afterwards, the rough pose of the remaining image in the data to be associated, that is, the second associated image, is inferred based on the target pose of the first associated image, and the inferred rough pose is used to associate with the reference data, that is, the image feature points that match between the second associated image and the second reference image in the reference data are determined, and finally all associated images in the associated data are associated with the reference image in the reference data. It can be understood that in this embodiment, the associated image and the reference image are associated, which can be understood as determining the image feature points that match between each frame of the associated image in the associated data and the corresponding reference image in the reference data, and the associated image and the reference image are associated through the matching image feature points.

[0187] In some embodiments, the second associated image may be an image adjacent to the first associated image in the to-be-associated data, or an image not adjacent to the first associated image. In this embodiment, the number of the second associated images may be one or more. When the number of the second associated images is multiple, the following two possible situations may be included:

[0188] (1) The second associated image is an image in the to-be-associated data that is adjacent to the first associated image. In this case, the first frame of the multiple frames included in the second associated image is adjacent to the last frame of the first associated image, and the images other than the first frame in the multiple frames may be adjacent or non-adjacent; or the last frame of the multiple frames included in the second associated image is adjacent to the first frame of the first associated image, and the images other than the last frame in the multiple frames may be adjacent or non-adjacent.

[0189] (2) The second associated image is an image in the to-be-associated data that is not adjacent to the first associated image, that is, the multiple frames of images included in the second associated image are not adjacent to the last frame of the first associated image, and the multiple frames of images may be adjacent or non-adjacent.

[0190] For the second associated image in the data to be associated, the rough pose of the second associated image can be calculated based on the target pose of the first associated image and the intra-pass relative pose of the second associated image, and then the image feature points that match between the second associated image and the second reference image can be determined based on the image feature points of the second associated image, the rough pose, the camera internal parameters for collecting the second associated image, and the initial pose of the second reference image in the reference data that has a co-viewing relationship with the second associated image, the image feature points of the second reference image and its corresponding three-dimensional point cloud data.

[0191] It can be understood that since the target pose of the first associated image has been determined, the target pose is a relatively accurate pose. Based on the target pose of the first associated image and the intra-pass relative pose of the second associated image, the rough pose of the second associated image can be inferred. Based on the rough pose, the second associated image can be matched with the image in the reference data to find the second reference image that has a common visual relationship with the second associated image, and the matching image feature points in the second associated image and the second reference image can be determined.

[0192] When there are other associated images that are not associated with the reference data in the data to be associated, the second associated image whose matching image feature points are determined in the previous round can be used as a new first associated image, and the target pose of the new first associated image is determined using the matching image feature points and the three-dimensional point cloud data corresponding to the image feature points in the second reference image that matches the image feature points. Then, a part of the other associated images that are not associated with the reference data in the data to be associated is taken out or all other remaining associated images are taken out as new second associated images, and the above steps are repeated to determine the image feature points that match between the new second associated image and the new second reference image based on the target pose of the new first associated image, the relative pose within the trip of the new second associated image, the image feature points of the new second associated image, the camera internal parameters for collecting the new second associated image, the initial pose of the new second reference image in the reference data, the image feature points and their corresponding three-dimensional point cloud data, etc. In this way, the image feature points of the second associated images other than the first associated image in the data to be associated and the second reference image in the reference data that match each other can be obtained, and finally the image feature points that match each frame of associated image in the data to be associated and the corresponding reference image in the reference data can be determined, thereby establishing an association relationship between the images in the data to be associated and the reference data.

[0193] In some embodiments, the intra-pass relative posture of the second associated image can be understood as the relative posture of the second associated image relative to the first frame image in the data to be associated, or the relative posture of the second associated image relative to other images in the data to be associated, for example, it can be the relative posture relative to the previous frame image of the second associated image.

[0194] The disclosed embodiment provides a data association method, which can select one of the collected data as reference data and the other collected data as data to be associated after acquiring two collected data of the same road. Using three-dimensional reconstruction technology, three-dimensional point cloud data corresponding to image feature points in the reference data is obtained. In order to align the pose of the data to be associated with the reference data, multiple frames of images can be selected from the data to be associated as first associated images, and multiple frames of first reference images having a co-viewing relationship with the first associated image can be selected from the reference data. Then, based on the initial pose of the first reference image, the image feature points on the first reference image and the three-dimensional point cloud data corresponding to the image feature points, the initial pose of the first associated image, the image feature points, the first relative pose of the first associated image, and the camera internal parameters of the first associated image, the image feature points that match between the first associated image and the first reference image and the target pose of the first associated image are determined after aligning the first associated image with the corresponding first reference image. Afterwards, the rough pose of the second associated image can be determined based on the target pose of the first associated image and the intra-pass relative pose of the second associated image other than the first associated image in the data to be associated. The image feature points that match between the second associated image and the second reference image can be determined based on the image feature points, rough pose, camera internal parameters for collecting the second associated image, the initial pose of the second reference image in the reference data that has a co-viewing relationship with the second associated image, image feature points, and three-dimensional point cloud data corresponding to the image feature points. All associated images in the data to be associated are processed in the same manner as above, so that the reference data and the associated data can be preliminarily associated and aligned. The above method provided by the embodiment of the present disclosure realizes the rapid association of two passes of collected data by first associating and aligning some images in the data to be associated with the reference data, and then associating and aligning other images in the data to be associated with the reference data based on the aligned partial images, thereby improving the efficiency of data association.

[0195] The data association device in this embodiment corresponds to the data association method described above. For specific details, please refer to the description of the data association method described above, which will not be repeated here.

[0196] Figure 4 It is a schematic diagram of the structure of a server suitable for implementing the data association method according to an embodiment of the present disclosure.

[0197] like Figure 4As shown, the server 400 includes a processing unit 401, which can be implemented as a processing unit such as a CPU, a GPU, an FPGA, and an NPU. The processing unit 401 can perform various processes in the embodiments of any of the above methods of the present disclosure according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 708 into a random access memory (RAM) 403. In RAM403, various programs and data required for the operation of the server 400 are also stored. The processing unit 401, ROM402, and RAM403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0198] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0199] In particular, according to an embodiment of the present disclosure, any method in the above referenced embodiment of the present disclosure may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing any method in the embodiment of the present disclosure. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411.

[0200] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0201] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0202] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0203] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A data association method, wherein: The method comprises: Acquire two rounds of collected data on the same road, wherein each round of collected data includes: multiple frames of images collected on the road, image feature points of each frame of image, and initial position and relative position of each frame of image within the round; From the two rounds of collected data, one round of collected data is selected as reference data, and the other round of collected data is selected as data to be associated; Using three-dimensional reconstruction technology, obtain three-dimensional point cloud data corresponding to the image feature points in the reference data; Acquire multiple frames of first reference images and first associated images having a common viewing relationship from the reference images included in the reference data and the associated images included in the to-be-associated data; Determine the image feature points that match between the first reference image and the first associated image, and the target pose of the first associated image based on the initial pose, image feature points, and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, and the intra-pass relative pose of the first associated image, and the intrinsic parameters of the camera used to acquire the first associated image; Obtaining a rough pose of a second associated image based on the target pose of the first associated image and the intra-pass relative poses of second associated images other than the first associated image in the to-be-associated data; Determine image feature points that match between the second associated image and the second reference image based on image feature points, a rough pose, an intrinsic parameter of a camera for acquiring the second associated image, and an initial pose, image feature points, and the three-dimensional point cloud data corresponding to the image feature points of a second reference image in the reference data that has a co-viewing relationship with the second associated image; If the to-be-associated data also includes an unprocessed associated image, determining a target pose of the second associated image based on image feature points matching the second associated image and the three-dimensional point cloud data corresponding to the image feature points; The step of taking the second associated image as a new first associated image, taking the unprocessed associated image as a new second associated image, returning the target pose based on the first associated image and the intra-pass relative pose of the second associated image other than the first associated image in the data to be associated, and obtaining the rough pose of the second associated image is continued until all the associated images in the data to be associated are processed.

2. The method according to claim 1, wherein: Based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, intra-pass relative pose of the first associated image and the intrinsic parameters of a camera used to acquire the first associated image, image feature points that match between the first reference image and the first associated image, and a target pose of the first associated image, the method includes: Based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, and relative pose within the first associated image, and the intrinsic parameters of the camera used to collect the first associated image, the pose of the first associated image is rigidly aligned to the first reference image to obtain a coarse pose of the first associated image; Based on the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the coarse pose of the first associated image, the image feature points, and the camera intrinsic parameters for acquiring the first associated image, the image feature points that match between the first reference image and the first associated image and the target pose of the first associated image are determined.

3. The method according to claim 2, wherein: Based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points and the intra-pass relative pose of the first associated image, and the camera intrinsic parameters for collecting the first associated image, the pose of the first associated image is rigidly aligned to the first reference image to obtain a coarse pose of the first associated image, including: Based on the initial position and posture of the first reference image, the image feature points and the three-dimensional point cloud data corresponding to the image feature points, and the initial position and posture of the first associated image and the image feature points, obtain image feature points having a rough matching relationship between the first associated image and the first reference image; Based on the image feature points with a rough matching relationship, the three-dimensional point cloud data of the image feature points, the camera intrinsic parameters for collecting the first associated image, and the intra-pass relative pose of the first associated image, it is determined that the rough pose of the first associated image is rigidly aligned to the first reference image.

4. The method according to claim 2 or 3, wherein: Determining image feature points that match between the first reference image and the first associated image and a target pose of the first associated image based on three-dimensional point cloud data corresponding to image feature points of the first reference image, a coarse pose of the first associated image, image feature points, and an intrinsic parameter of a camera used to acquire the first associated image, including: Based on the image feature points in the first associated image, the image feature points of the first reference image and the three-dimensional point cloud data corresponding to the image feature points, the coarse pose of the first associated image and the camera intrinsic parameters for acquiring the first associated image, the three-dimensional point cloud data corresponding to the image feature points in the first reference image are projected onto the first associated image, image feature points whose reprojection errors are greater than a set value are filtered out, and image feature points whose matching relationship is accurate between the first reference image and the first associated image are determined; The target posture of the first associated image is determined based on the image feature points with accurate matching relationship and the three-dimensional point cloud data corresponding to the image feature points of the first reference image.

5. The method according to claim 3, wherein: After determining that the pose of the first associated image is rigidly aligned to the first reference image based on the image feature points having a rough matching relationship, the three-dimensional point cloud data of the image feature points, the camera intrinsic parameters for collecting the first associated image, and the intra-pass relative pose of the first associated image, the rough pose of the first associated image includes: Determining the candidate pose of each first associated image after the pose of each first associated image is rigidly aligned with the corresponding first reference image based on the image feature points having a coarse matching relationship, the three-dimensional point cloud data corresponding to the image feature points, and the camera intrinsic parameters for acquiring the first associated image; Determine a first-frame optimized pose of a first frame of the first associated images based on the candidate poses of each of the first associated images and the intra-pass relative poses of the first associated images; Based on the first frame optimized pose and the intra-pass relative pose of the first associated image, a coarse pose of the first associated image is determined.

6. The method according to claim 5, wherein: The step of determining the first frame optimized pose of the first frame image in the first associated images based on the candidate poses of each of the first associated images and the intra-pass relative poses of the first associated images comprises: Determine a plurality of first-frame candidate poses of a first frame image in the first associated images based on each of the candidate poses of each of the first associated images and the intra-pass relative poses of the first associated images; Obtaining multiple reference poses of each of the first associated images by using multiple first frame candidate poses and the intra-pass relative poses of the first associated images; Based on the deviation between the candidate pose of each of the first associated images and the corresponding reference pose, the first frame optimized pose of the first frame image is screened out from the plurality of first frame candidate poses.

7. The method according to claim 4, wherein: Based on the image feature points in the first associated image, the image feature points of the first reference image and the three-dimensional point cloud data corresponding to the image feature points, the coarse pose of the first associated image and the camera intrinsic parameters for acquiring the first associated image, the three-dimensional point cloud data corresponding to the image feature points in the first reference image are projected onto the first associated image, image feature points whose reprojection errors are greater than a set value are filtered out, and image feature points with accurate matching relationships between the first reference image and the first associated image are determined, including: Re-determining image feature points having a rough matching relationship between the first associated image and the first reference image based on image feature points in the first associated image, image feature points in the first reference image, three-dimensional point cloud data corresponding to the image feature points, and the rough pose; Based on the re-determined image feature points with a rough matching relationship, the three-dimensional point cloud data corresponding to the image feature points, the camera intrinsic parameters for collecting the first associated image, and the rough pose of the first associated image, projecting the three-dimensional point cloud data corresponding to the re-determined image feature points with a rough matching relationship onto the first associated image; Determine a reprojection error between a projection point on the first associated image and the image feature point having a rough matching relationship re-determined in the first associated image; The image feature points whose reprojection errors are greater than the set value are filtered out from the re-determined image feature points with a rough matching relationship, so as to obtain image feature points with an accurate matching relationship between the first reference image and the first associated image.

8. The method according to claim 4, wherein: Determining the target pose of the first associated image based on the image feature points with accurate matching relationship and the three-dimensional point cloud data corresponding to the image feature points of the first reference image includes: Based on the three-dimensional point cloud data corresponding to the image feature points with accurate matching relationship and the image feature points of the first reference image, the pre-optimization pose of the first associated image is calculated; The pre-optimization pose of the first associated image and the intra-pass relative pose of the first associated image are used as constraints, and the pre-optimization pose of the first associated image is optimized by constructing a pose optimization graph to obtain a target pose of the first associated image.

9. The method according to any one of claims 1-3, 5-8, wherein: Determining image feature points that match between the second associated image and the second reference image based on image feature points, a rough pose, an intrinsic parameter of a camera for acquiring the second associated image, an initial pose, an image feature point, and the three-dimensional point cloud data corresponding to the image feature point of a second reference image in the reference data that has a co-viewing relationship with the second associated image, comprising: Determine image feature points having a rough matching relationship between the second associated image and the second reference image by using the image feature points and the rough pose of the second associated image as well as the initial pose, image feature points of the second reference image and the three-dimensional point cloud data corresponding to the image feature points; Based on the image feature points with a rough matching relationship, the three-dimensional point cloud data corresponding to the image feature points of the second reference image, the rough pose of the second associated image and the camera intrinsic parameters for acquiring the second associated image, the three-dimensional point cloud data corresponding to the image feature points with a rough matching relationship are projected onto the second associated image, and from the image feature points with a rough matching relationship, image feature points whose reprojection errors are greater than a set value are filtered out to obtain image feature points with an accurate matching relationship between the second reference image and the second associated image.

10. A data association device, wherein: The data association device comprises: The first acquisition module is configured to acquire two trips of collected data on the same road, wherein each trip of collected data includes: multiple frames of images collected on the road, image feature points of each frame of image, and an initial position and relative position of each frame of image within the trip; A selection module is configured to select one of the two rounds of collected data as reference data and the other round of collected data as data to be associated; A first determination module is configured to obtain three-dimensional point cloud data corresponding to the image feature points in the reference data using three-dimensional reconstruction technology; A second acquisition module is configured to acquire a plurality of frames of first reference images and first associated images having a common viewing relationship from the reference images included in the reference data and the associated images included in the to-be-associated data; A second determination module is configured to determine the image feature points that match between the first reference image and the first associated image, and the target pose of the first associated image based on the initial pose, image feature points and the three-dimensional point cloud data corresponding to the image feature points of the first reference image, the initial pose, image feature points, the intra-pass relative pose of the first associated image and the camera intrinsic parameters for acquiring the first associated image; A third determination module is configured to obtain a rough pose of a second associated image based on the target pose of the first associated image and an intra-pass relative pose of a second associated image other than the first associated image in the to-be-associated data; a fourth determination module, configured to determine image feature points that match between the second associated image and the second reference image based on image feature points, a coarse pose, an intrinsic parameter of a camera for acquiring the second associated image, an initial pose, an image feature point, and the three-dimensional point cloud data corresponding to the image feature point of a second reference image in the reference data that has a co-viewing relationship with the second associated image; a fifth determination module, configured to determine a target pose of the second associated image based on image feature points matching the second associated image and three-dimensional point cloud data corresponding to the image feature points if the to-be-associated data also includes an unprocessed associated image; The sixth determination module is configured to use the second associated image as a new first associated image and the unprocessed associated image as a new second associated image, return the target posture based on the first associated image and the intra-pass relative posture of the second associated image other than the first associated image in the data to be associated, and continue to execute the step of obtaining the rough posture of the second associated image until all the associated images in the data to be associated are processed.

11. A server, wherein: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images