Method for determining the position and orientation of earth moving machinery in a work site

Abstract

A method for determining a position and orientation of an earthmoving machine in a work site includes providing at least one positioning camera on the earthmoving machine, determining a plurality of distinguishable features from a work site environment as a plurality of map feature points, taking a two-dimensional positioning image with the at least one positioning camera, detecting the distinguishable features from the two-dimensional positioning image as navigation feature points, and matching identification information of the navigation feature points with identification information of the plurality of map feature points to determine the position and orientation of the earthmoving machine in the work site.

Description

Technical Field

[0001] This invention relates to a method for determining the position and orientation of earthmoving machinery on a construction site. Background Technology

[0002] Different types of earthmoving machinery may be used at different earthmoving construction sites, for example, to move soil or rock materials from one location to another. Examples of such sites include substructure construction sites, building construction sites, and road construction sites. Examples of such earthmoving machinery include excavators and bulldozers.

[0003] Earthmoving machinery and its tools should be able to be positioned and operated very accurately on the site to correctly perform the designed operations. Information regarding the precise location and orientation of the earthmoving machinery and its tools should be provided to the operator so that the operator can use this information when controlling the tools and machinery. This precise location and orientation information is particularly important when used in semi-automatic or fully automatic earthmoving machinery (i.e., where the earthmoving machinery is not continuously controlled by the operator for at least a period of time, and therefore any possible misalignment of the machinery or its tools is not immediately corrected by the operator).

[0004] Typically, for example, automatic positioning of machinery can be based on a satellite positioning system GNSS (Global Navigation Satellite System), such as GPS (US), GLONASS (RU), Galileo (EU), or Compass (CN). Alternatively, positioning of earthmoving machinery can be provided by a total station located at the construction site.

[0005] However, not every construction site or every piece of earthmoving machinery has a satellite positioning system that is accurate enough, or the connection to the satellite may be interrupted due to obstacles or objects reaching above the earthmoving machinery. Furthermore, setting up a total station-based positioning system on a construction site can be time-consuming and labor-intensive, especially if the system needs to be removed from the site daily or multiple times a day.

[0006] Therefore, an alternative positioning solution is needed. Summary of the Invention

[0007] The purpose of this invention is to provide a new method for determining the location and orientation of earthmoving machinery on a construction site.

[0008] The present invention is characterized by the features of the independent claims.

[0009] This invention is based on the idea of ​​multiple distinguishable features to be used as unique map feature points for site identification. Each of these map feature points has identification information and location data. The identification information includes at least a unique identifier, similar to a feature descriptor, and the location data includes at least one of the following: its three-dimensional position in the site environment or two-dimensional image coordinates with an image identifier. Map feature points with identification information and location data are used to implement a visual positioning method for working machines, wherein the positioning error is kept at a controllable level, i.e., at an acceptable level, rather than accumulating over time. In georeferencing, particular attention is paid to the multiple distinguishable features to be used as unique map feature points.

[0010] The advantage of this invention is the accurate feature point map, which is used to determine the position and orientation of earthmoving machinery on the construction site.

[0011] Some embodiments of the invention are disclosed in the dependent claims.

[0012] According to one embodiment of a method for determining the position and orientation of earthmoving machinery on a construction site, the method includes setting at least one positioning camera on the earthmoving machinery, the at least one positioning camera having: a defined intrinsic parameter set, wherein the intrinsic parameter set defines the formation of each image pixel from a real-world view and the determined position and orientation in the machine coordinate system; and determining a plurality of distinguishable features from the construction site environment as a plurality of map feature points, wherein the plurality of map feature points have at least identification information and location data, wherein the identification information includes at least a unique identifier, and the location data includes at least one of a three-dimensional position on the construction site or two-dimensional image coordinates having an image identifier; Furthermore, a two-dimensional positioning image is captured using at least one positioning camera; distinguishable features are detected from the two-dimensional positioning image as navigation feature points, and the two-dimensional image coordinates and identification information of each navigation feature point are determined; and the identification information of the navigation feature points is matched with the identification information of at least one of the plurality of map feature points or detected mismatched feature points; wherein, the position and orientation of the earthmoving machinery on the construction site are determined based on at least one of the following position data: map feature points or detected mismatched feature points, the two-dimensional image coordinates of the corresponding matched navigation feature points, and the determined intrinsic parameter set and the determined position and orientation of each corresponding positioning camera in the machine coordinate system.

[0013] According to one embodiment of the method, the position and orientation of earthmoving machinery in a site coordinate system are determined, the method comprising:

[0014] The feature point map is generated using the following method:

[0015] Multiple distinguishable features are determined from the construction site environment as multiple map feature points by taking multiple two-dimensional images of the construction site environment by at least one camera with a defined set of intrinsic parameters, wherein the set of intrinsic parameters defines the formation of each image pixel from the real-world view.

[0016] Detect at least one of the following from each of the plurality of two-dimensional construction site environment images:

[0017] Distinguishing features are used to associate each distinguishable feature with the corresponding identifier information and image coordinates of a two-dimensional construction site environment image, as detected feature points; or

[0018] The distinguishable features of the reference feature points are preset in the construction site and associated with the identification information and location data in the construction site coordinate system. They are used to additionally associate each reference feature point with the image coordinates of the corresponding two-dimensional construction site environment image as a detected reference feature point.

[0019] Create at least one set of images using the following method:

[0020] By having at least one of the following five elements, site environment images are bound together: detected feature points or detected reference feature points that are matched as identical points in the site environment images to be bound.

[0021] Each set of images contains at least two site environment images and at least three detected reference feature points that are matched as identical points; and

[0022] By utilizing the identification information and location data of the at least three detected reference feature points, the position and orientation of each accepted image group are determined, and the identification information and location data of detected feature points matched as identical points are determined; thereby

[0023] The detected feature points that are matched with the same point and the detected reference feature points that are matched with the same point are determined as map feature points for the feature point map;

[0024] Among them, the plurality of map feature points in the feature point map have at least identification information and location data, wherein,

[0025] The identification information includes at least a unique identifier that identifies distinguishable features; and the location data includes the three-dimensional location in the site coordinate system.

[0026] At least one positioning camera is installed on the earthmoving machinery, the at least one positioning camera having a defined set of intrinsic parameters and a defined position and orientation in the machine coordinate system;

[0027] Provide feature point maps for earthmoving machinery;

[0028] Two-dimensional positioning images are captured using at least one positioning camera;

[0029] Distinguishing features are detected from the two-dimensional positioning image as navigation feature points, and the two-dimensional image coordinates and identification information of each navigation feature point are determined; and

[0030] The identification information of navigation feature points is matched with the identification information of map feature points on the feature point map; whereby...

[0031] The position and orientation of earthmoving machinery in the site coordinate system are determined based on at least the following:

[0032] Location data of matched map feature points;

[0033] The corresponding two-dimensional image coordinates of the matched navigation feature points; and

[0034] A defined set of intrinsic parameters and the defined position and orientation of each corresponding positioning camera in the mechanical coordinate system.

[0035] According to one embodiment of the method, the step of determining multiple distinguishable features as multiple map feature points from a construction site environment includes capturing multiple two-dimensional construction site environment images by at least one camera having a defined set of intrinsic parameters, and detecting at least one of the following from each of the multiple two-dimensional construction site environment images: a distinguishable feature, used to associate each distinguishable feature with the identification information and image coordinates of the corresponding two-dimensional construction site environment image as a detected feature point; or a reference feature point, wherein the reference feature point is pre-set in the construction site and associated with identification information and location data, used to additionally associate each reference feature point with the image coordinates of the corresponding two-dimensional construction site environment image as a detected reference feature point; and by having at least one of the following, by having the construction site environment images... This binding creates at least one set of images: detected feature points or detected reference feature points that match the same point in the site environment images to be bound; and accepts each set of images containing at least two site environment images and at least three detected reference feature points that match the same point; by utilizing the identification information and location data of the at least three detected reference feature points, the position and orientation of each accepted set of images are determined, and the identification information and location data of the detected feature points that match the same point are determined; wherein at least one of the following is determined as multiple map feature points: detected feature points that match the same point and have determined identification information and location data, or detected reference feature points that match the same point and are associated with identification information and location data.

[0036] According to one embodiment of the method, the step of determining multiple distinguishable features from a construction site environment as multiple map feature points includes capturing multiple two-dimensional construction site environment images using at least one camera, the at least one camera having a defined set of intrinsic parameters and a defined position in a construction site coordinate system; detecting distinguishable features from each of the multiple two-dimensional construction site environment images, and determining the distinguishable features having image coordinates of the corresponding two-dimensional construction site environment image as detected feature points; creating at least one set of images having detected feature points that match as identical points in the construction site environment images; and determining identification information and location data for each set of images for matching detected feature points that match as identical by utilizing the position of the at least one camera in the construction site coordinate system and each corresponding two-dimensional construction site environment image, wherein the detected feature points that match as identical with the determined identification information and location data are determined as multiple map feature points.

[0037] According to one embodiment of the method, in the step of determining multiple distinguishable features from the construction site environment as multiple map feature points, the map feature points are determined at least in part based on data retrieved from an earthwork information model, wherein the earthwork information model is based on at least one of: a geospatial information system (GIS), building information modeling (BIM), infrastructure building information modeling (I-BIM), civil information modeling (CIM), or a smart city platform.

[0038] According to one embodiment of the method, the step of determining multiple distinguishable features from the site environment as multiple map feature points further includes determining additional map feature points at least in part based on data retrieved from an earthwork information model, wherein the earthwork information model is based on at least one of: a geospatial information system (GIS), building information modeling (BIM), infrastructure building information modeling (I-BIM), civil information modeling (CIM), or a smart city platform.

[0039] According to one embodiment of the method, in the step of determining multiple distinguishable features from the site environment as multiple map feature points, the map feature points are determined as a combination set of map feature points determined by the method disclosed in at least two of claims 1, 2 or 3.

[0040] According to one embodiment of the method, the steps of determining the location and orientation of each received image group and determining identification information and location data for detected feature points matched as the same further include determining a static rating for the detected feature points matched as the same by utilizing the identification information and location data of at least three detected reference feature points, wherein at least one of the following is determined as a plurality of map feature points: detected feature points matched as the same with determined identification information, location data and static rating, or detected reference feature points associated with identification information and location data.

[0041] According to one embodiment of the method, the step of determining identification information and location data for each image group to match the same detected feature points further includes determining a static rating for the same detected feature points by utilizing the position of the at least one camera in the site coordinate system and each corresponding two-dimensional site environment image, wherein the same detected feature points with the determined identification information, location data and static rating are determined as multiple map feature points.

[0042] According to one embodiment of the method, the method further includes determining an overall rating, the overall rating including at least one of a static rating or a dynamic rating, and determining a dynamic rating individually for each of a plurality of determined map feature points with respect to each positioning camera, wherein the previously determined position and orientation of each of the at least one positioning camera are taken into account.

[0043] According to one embodiment of the method, the overall rating is at least one of the following: a static rating or a dynamic rating, and the overall rating is at least two-tiered.

[0044] According to one embodiment of the method, if the number of multiple map feature points exceeds a threshold, at least the map feature points with the lowest overall rating are discarded, wherein the threshold is determined by at least one of manual or automatic methods.

[0045] According to one embodiment of the method, the step of determining the position and orientation of earthmoving machinery on the construction site is further based on tracking navigation feature points between consecutive positioning images, wherein the navigation feature points tracked between consecutive positioning images indicate changes in the position and orientation of the positioning camera.

[0046] According to one embodiment of the method, the method further includes determining the location and orientation of earthmoving machinery in an earthmoving information model, wherein the earthmoving information model is based on at least one of: Geospatial Information System (GIS), Building Information Modeling (BIM), Infrastructure Building Information Modeling (I-BIM), Civil Information Modeling (CIM), and a smart city platform.

[0047] According to one embodiment of the method, the method further includes tools for controlling earthmoving machinery to create structures described in an earthmoving information model.

[0048] According to one embodiment of the method, the method further includes generating as-built data that will accompany the earthwork information model.

[0049] According to one embodiment of the method, the method further includes transmitting as-built data to at least one of the following: a site information management system or machinery operating at the site. Attached Figure Description

[0050] In the following description, the invention will be described in more detail with reference to the accompanying drawings and by means of preferred embodiments, in which:

[0051] Figure 1 A schematic side view of an excavator at a construction site is shown.

[0052] Figure 2a An embodiment of a method for determining the location and orientation of earthmoving machinery on a construction site is illustrated schematically.

[0053] Figure 2b A system for determining the position and orientation of earthmoving machinery on a construction site is schematically illustrated.

[0054] Figure 3 The illustration schematically shows a method and system for providing feature point maps for a construction site;

[0055] Figure 4 An example of creating a feature descriptor for a single pixel in an image is illustrated.

[0056] Figure 5a , Figure 5b , Figure 5c and Figure 5d An embodiment of determining map feature points for creating a feature point map is illustrated schematically;

[0057] Figure 6 Another embodiment of determining map feature points for creating a feature point map is illustrated schematically;

[0058] Figure 7a , Figure 7b and Figure 7c Combination Figure 2a and Figure 2b An embodiment for determining the position and orientation of machinery on a construction site is illustrated schematically;

[0059] Figure 8 Some earthwork engineering information models are schematically disclosed; and

[0060] Figure 9 An embodiment related to the visual odometry process is illustrated schematically.

[0061] For clarity, the accompanying drawings illustrate some embodiments of the invention in a simplified manner. In the drawings, similar reference numerals identify similar elements. Detailed Implementation

[0062] Figure 1 This is a schematic side view of excavator 1 at site 14, where excavator 1 will be operated. Excavator 1 is an example of earthmoving machinery, and the solutions disclosed herein for determining the position and orientation of earthmoving machinery in site 14 can be applied in conjunction with it. Site 14 includes at least one three-dimensional area or space, i.e., the working environment WE, where the activities to be performed by the earthmoving machinery will take place. Depending on the nature or characteristics of site 14, site 14 may include one or more surrounding areas or spaces in addition to the area or space where the activities will be performed, which may affect the operations to be performed in the area or space where the activities will take place and / or the operations occurring in the area or space where the activities will take place may affect the surrounding areas or spaces. Especially in urban environments, earthmoving machinery may also need to work temporarily outside the actual or officially designated site. Therefore, in this specification, site 14 may also extend some distance beyond the area and / or space designated as actual or officially designated site 14, such as extending beyond several meters.

[0063] The excavator 1 includes a movable bracket 2, which comprises a lower bracket 2a (i.e., lower support 2a) and an upper bracket 2b. The lower support 2a includes tracks, but alternatively, wheels. The upper bracket 2b is connected to the lower support 2a via a rotation axis 3. Therefore, the upper bracket 2b can rotate relative to the lower support 2a about a rotation axis 4, as schematically shown by arrow R. The rotation axis 4 coincides with the central axis of the rotation axis 3.

[0064] The excavator 1 further includes a boom 5 connected to the upper bracket 2b, wherein the boom 5 is arranged to rotate together with the upper bracket 2b. The boom 5 may include at least a first boom component 5a. The boom 5 may also include other boom components, such as a second boom component 5b. The boom 5 can be raised and lowered relative to the upper bracket 2b, as schematically shown by arrow L.

[0065] The second boom assembly 5b can be connected to the first boom assembly 5a via a joint 6, thereby allowing the second boom assembly 5b to rotate about the first boom assembly 5a, as schematically shown by arrow T6. A working tool, in this case a bucket 7, is located at the distal end of the second boom assembly 5b, and a joint 8 can be provided between the bucket 7 and the second boom assembly 5b, thereby allowing the bucket 7 to rotate about the second boom assembly 5b, as schematically shown by arrow T8. For example, in conjunction with joint 8, a joint or mechanism may also be provided that allows the bucket to tilt in a lateral direction.

[0066] The bracket 2 may have a control room 9 for the operator 10 of the excavator 1. For example, the control room 9 may be provided with a movable arrangement, thereby allowing the vertical position of the control room 9 to be adjusted relative to the bracket 2.

[0067] The excavator 1 further includes at least one control unit 11, which is configured to control the operation of the excavator 1, such as the operation of the bracket 2, the boom 5 and the bucket 7, in response to a received control action.

[0068] If the excavator 1 is intended to be able to utilize a satellite positioning system GNSS (Global Navigation Satellite System), the excavator 1 may further include multiple satellite receiving devices, such as one or more antennas 12, with satellites shown schematically indicated by reference numeral 13. For example, the one or more antennas 12 may be arranged on the upper bracket 2b.

[0069] The excavator 1 may further include multiple sensor SMUs or multiple sets of sensor SMUs, such as inertial measurement units (IMUs), for determining, for example, the position and / or orientation and / or tilt and / or azimuth of the machinery. The sensor or set of sensors may be at least one of the following: a gyroscope, accelerometer, inclinometer, magnetic compass, angle sensor, position sensor, pendulum, level measuring device, and any other sensor suitable for determining at least one of the position, orientation, tilt, or azimuth of at least one object attached to each other, such as a camera sensor, stereo camera, laser receiver / detector, or lidar. If the excavator 1 is equipped with at least one antenna 12 capable of utilizing a GNSS satellite positioning system, the at least one sensor may also be a satellite-based compass. The excavator 1 may also be equipped with a sensor for measuring the distance traveled by the excavator 1, which may be an internal or external sensor of the inertial measurement unit (IMU).

[0070] Figure 1 Several coordinate systems are also schematically disclosed, which can be used in the disclosed solution to determine the position and orientation of excavator 1 in site 14. Figure 1A machine coordinate system (MCS) is disclosed, which is attached to a point in the excavator 1, and allows the position and orientation of an object relative to that point to be determined within the MCS. Additionally, Figure 1 A site coordinate system (WCS) is disclosed, which is attached to a point in site 14 and allows the determination of the position and orientation of an object relative to that point within the WCS. Furthermore, Figure 1 A World Coordinate System (WDCS) is disclosed, which is attached to a point in the real world and allows the position and orientation of an object relative to that point to be determined in the WDCS.

[0071] Figure 2a An embodiment of a method for determining the position and orientation of earthmoving machinery in site 14 is illustrated schematically. Figure 2b A system-level diagram illustrating the determination of the location and orientation of earthmoving machinery at site 14 is shown schematically. The excavator 1 disclosed above is merely one example of earthmoving machinery, to which the solutions disclosed herein can be applied. The solutions disclosed herein can also be applied to other earthmoving machinery, such as bulldozers, wheel loaders, road rollers, backhoe excavators, dump trucks, cranes, mobile cranes, freighters, harvesters, etc. Therefore, in the following description, the earthmoving machinery may also be referred to as "machinery 1".

[0072] The method for determining the position and orientation of machine 1 in site 14 includes setting at least one positioning camera PCA on machine 1, i.e., one or more positioning camera PCAs. Each of the one or more positioning camera PCAs has a defined set of intrinsic parameters and a defined position and orientation in the machine coordinate system MCS.

[0073] A defined set of intrinsic parameters defines the formation of each image pixel from a real-world view for an image to be captured by the corresponding positioning camera PCA. This defined set of intrinsic parameters includes parameters that define the operation of the camera. These parameters may include, for example, image center, focal length, skew, and lens distortion. The defined set of intrinsic parameters describes at which image pixel in the captured image is a specific point in the real world. The defined set of intrinsic parameters for each positioning camera PCA can be obtained beforehand via laboratory calibration or later during the “Structure from Motion” process, discussed in more detail later.

[0074] In the method described, the determined position and orientation of the at least one positioning camera PCA in the machine coordinate system MCS must be known. This can be determined when each positioning camera PCA is mounted on the machine 1. It can be a combination of manual measurement and sensor measurement, or entirely by sensor measurement. In each case, it is important that the determined position and orientation of each positioning camera PCA relative to the machine coordinate system MSC be measured as accurately as possible, considering its origin and axes (x, y, z). Measurements by sensors can be determined, for example, by at least one sensor or measuring device external to the machine 1 and used only when the positioning camera PCA is mounted on the machine 1, or by at least one sensor SMU mounted in the machine 1 and / or by at least one sensor SMC mounted in the positioning camera PCA. The at least one sensor SMC in the positioning camera PCA (i.e., one or more sensor SMCs) and the at least one sensor SMU mounted on the machine 1 (i.e., one or more sensor SMUs) can be used to determine, for example, the position and / or orientation and / or tilt and / or azimuth of the positioning camera PCA. Regardless of the measurement of the camera's position and orientation in the machine coordinate system (MCS), the system used to determine the position and orientation of the earthmoving machinery on the site requires the precise position and orientation of each of the at least one positioning camera PCA in the MCS. During operation of the machinery 1, the position and orientation of the positioning camera PCA in the MCS can also be determined or verified by using at least one sensor SMU in the machinery 1 and / or at least one sensor SMC in the positioning camera PCA to ensure that the position and / or orientation of the at least one positioning camera PCA remains as determined by the system, or if not, can be re-determined by the system if it changes. Similarly, if the position or orientation of the positioning camera PCA has changed, the operator of the machinery can be notified. The at least one sensor SMC in the positioning camera PCA can be, for example, at least one of the following: a gyroscope, accelerometer, inclinometer, magnetic compass, angle sensor, position sensor, pendulum, level measuring device, or any other sensor or measuring device suitable for measuring position and / or orientation or changes in position and / or orientation.

[0075] The method for determining the position and orientation of machinery 1 in site 14 further includes creating or generating a feature point map, i.e., a navigation map, which forms a representation of map feature points in site 14, said representation of map feature points to be used to determine the position and orientation of machinery 1 operating in site 14. Therefore, this navigation map is a pre-created feature point map of site 14, which will then be used for real-time navigation of machinery 1 in site 14. Some alternative solutions for creating feature point maps will be discussed in more detail later in this specification.

[0076] The position and orientation of machine 1 in site 14 are determined at least based on information available in the feature point map and the determined set of intrinsic parameters, as well as the determined position and orientation of each corresponding positioning camera PCA in the machine coordinate system MCS. This will be discussed later in conjunction with... Figure 7a , Figure 7b and Figure 7c And related explanations, considering Figure 2a and Figure 2b The steps for determining the position and orientation of machine 1 in site 14 will be discussed in more detail.

[0077] The necessary data processing for determining the location and orientation of machine 1 on site 14 can be performed using data processing resources available only within machine 1 (like control unit 11), data processing resources available only outside machine 1 (such as data processing resources reserved for a site information management system (such as a site server or computer), or data processing resources arranged to provide a cloud service for a site information management system), or data processing resources available both inside and outside machine 1. Similarly, the feature point map used to determine the location and orientation of machine 1 on site 14 can be located in at least one of machine 1, the site information management system, or the cloud service. Necessary data transmission can be achieved through dedicated and / or public data transmission networks available on site 14. Figure 2b In the middle, the control unit 11, which can be replaced by the construction site information management system or cloud services, is... Figure 2b The middle is very schematically drawn with dashed lines and surrounded Figure 2b The boxes representing items in the system.

[0078] Returning to the topic of creating or generating feature point maps, which has already been briefly discussed above, Figure 3A method for providing a feature point map for a construction site 14 is illustrated schematically. An embodiment of the method includes determining multiple distinguishable features from the construction site environment (WE) as multiple map feature points (MFPs), each of the multiple map feature points (MFPs) having at least identification information and location data. The identification information includes at least one unique identifier for the corresponding map feature point (MFP), while the location data includes at least one of the following: the three-dimensional position of the corresponding map feature point (MFP) in the construction site coordinate system (WCS) or two-dimensional image coordinates with an image identifier. The unique identifier may include at least some kind of index number for a database and at least one feature descriptor. The feature descriptors identify the map feature points (MFPs) from each other and thus can specify each distinguishable feature from each other. Typically, the map feature points (MFPs) use a feature descriptor, an ARUCO code, or an index number for a checker marker as a unique identifier. In other words, in the creation of the feature point map, multiple images are taken from the site environment WE by at least one camera, and these images are then examined or analyzed to find distinguishable features DF in these images. The distinguishable features DF are considered suitable for providing map feature points MFP from the site environment WE to form a feature point map, i.e., a pre-created navigation map for real-time navigation of machinery 1 in site 14.

[0079] Later, we will consider in more detail some different solutions for determining the Map Feature Points (MFP) used to form the feature point map. Before that, we will consider some technical aspects related to discovering and analyzing distinguishable features (DFs) in images taken from the site environment (WE).

[0080] To create a feature point map, i.e., a navigation map, multiple images are captured from the site environment WE at different locations of the camera / multiple cameras that took the images. At least one camera here can be at least one positioning camera (PCA) applied to some kind of machinery 1, or some other camera (CA) applied to site 14, for example, a camera applied to a drone moving within site 14. This process of capturing images from site 14 is reminiscent of aerial imaging or aerial imagery that allows for the reconstruction of a three-dimensional scene of the site environment WE. However, the perspective of the captured images is preferably substantially the same as the perspective of at least one positioning camera (PCA) applied to machinery 1 when determining the position and orientation of machinery 1 within site 14. Images are captured with a certain amount of overlap to combine or bind the images together to form a plurality of image groups or at least one image group GIM. It should be noted that each image must overlap sufficiently with at least one other image in the image group GIM, but not with all other images in that image group GIM. The overlap of the images can vary depending on the specific circumstances, for example, between 40% and 80%. If the imaged area contains many static entities, such as buildings with numerous features, the overlap can be small. Conversely, if the imaged area contains many moving or variable objects, such as gravel, small trees, shrubs, or lawns, or even smooth white walls, the overlap should be much larger. The greater the overlap, the more likely the image is to be successfully bound, and the denser the feature point map can be created.

[0081] If an inertial measurement unit (IMU) and / or GNSS are available, the image data for each captured image can be augmented with the attitude data and / or GNSS position data for each image. These can be used to create a feature point map during the "motion structure" workflow, a generally known technique for 3D scene reconstruction, and involve searching for features from images, feature matching between images, motion recovery and triangulation using the camera's intrinsic parameters, and optionally finding the optimal model parameters for scene reconstruction through an optimization task called bundle adjustment as key elements.

[0082] Bundle adjustment is an optimization task in which the parameters required for 3D scene reconstruction are solved. These parameters can be extrinsic / external parameters, such as the rotation and translation of the image relative to other images, the 3D coordinates of common feature points in the images, and intrinsic camera parameters. Extrinsic camera parameters can also be, for example, the camera's position and orientation in the World Wide Web Convergence Table (WDCS) or the site-specific WCS. Therefore, the mathematical model forming part of bundle adjustment can vary depending on the number of parameters applied in the model. Thus, it is a design choice that defines the problem to be solved. Furthermore, each parameter to be solved or optimized in the model can be assigned an initial value with specified initial values, while some parameters in the model can have pre-known fixed values. The purpose of bundle adjustment is to optimize all the selected model parameters to be optimized, such as extrinsic image parameters related to information about the features presented in the captured images. If the intrinsic and extrinsic camera parameters discussed above are known, bundle adjustment is not necessary. It may not be meaningful to include all matched detected feature points (DFPs) in bundle adjustment optimization. Discarded detected feature points (DFPs) can be reconstructed in the final triangulation after bundle adjustment. Therefore, when navigating using a PnP-based method, these detected feature points (DFPs) can be utilized. Furthermore, since the extrinsic parameters of the image are solved in the beam adjustment, in the embodiments discussed later, even mismatched detected feature points (DFPs-NM) residing on the image can be used when determining the position and orientation of the machine in real time, and thus can be stored using MFPs without three-dimensional coordinates. It should be noted that when referring to position and / or orientation / attitude, intrinsic parameters of the image, extrinsic parameters of the image, or parameters of the image, these are analogous to the camera parameters at the time the image was captured. Therefore, for example, if the camera's intrinsic and extrinsic parameters are known at the time the image was captured, these parameters can be used as the intrinsic and extrinsic parameters of the image captured by the camera; if the camera's intrinsic and extrinsic parameters are not known at the time the image was captured, they will be solved in the beam adjustment and will also be discussed as the intrinsic and extrinsic parameters of the image. Alternatively, if only the intrinsic camera parameters are known, they can also be fed into the beam adjustment as known information. As noted above, the bundle adjustment process is a well-known technique related to 3D scene reconstruction. However, for example, the published bundle adjustment (A Modern Synthesis, Vision Algorithms: Theory & Practice, B. Triggs, A. Zisserman & R. Szeliski (eds.), Springer-Verlag LNCS 1883, 2000) discloses the bundle adjustment process in more detail.

[0083] Typically, when images are taken from a construction site environment, no image pose and / or GNSS location data are available. For example, GNSS may be unavailable in tunnels, under bridges, or in urban environments with tall buildings, or the camera CA / PCA taking the images may not be connected to a positioning system that will determine the position and / or orientation of the camera CA / PCA. In such cases, a network of ground control points can be applied. Ground control points are reference points or reference feature points preferably uniformly distributed across the construction site 14. Because these reference points need to be identified from the images, they are equipped with specific markers to allow for their identification. Ground control points can be, for example, square grid markers or Aruco markers, the latter generally being more accurately detected from images. If square markers are used, the markers do not need to be unique; that is, they can all be identical, thus specifying the corresponding marker based on the location of each marker. The grid markers can be detected in the image, but their actual recognition is done, for example, by using computer-readable numbers on the markers, through background-based recognition (i.e., what the surrounding objects are visible in the image), or later in a 3D scene using a combination of angles between points in the reconstructed 3D coordinate system of the image set and the distance of the markers.

[0084] For each image, GNSS position data and, possibly, attitude data, or alternatively, a ground control point network, are used to create a feature point map, i.e., a navigation map, that appropriately incorporates the site environment scene into the site coordinate system (WCS). Alternatively, the attitude data and GNSS position data for each image can be used together with the ground control point network to create the feature map, thus bringing the site environment scene into the WCS at an appropriate scale. When bundle adjustment is applied, the coordinate transformation from the reconstructed 3D site environment scene to the world WDCS or site WCS coordinate system can be solved in a separate optimization task after bundle adjustment. The covariance matrix, which depicts the accuracy information for the detected features, can also be solved during bundle adjustment.

[0085] To reconstruct a 3D scene from captured images of a construction site, it's necessary to find common feature points from multiple images. The 3D coordinates of these common feature points can then be calculated. The more common feature points found in the captured images, the higher the accuracy of the reconstruction.

[0086] In the captured images, features in the 3D construction site environment WE are depicted on a 2D plane. The 2D position of these features in the image is determined by 2D image coordinates, i.e., image pixel coordinates. Finding the same features from several images can be done completely automatically without manually picking common object points in the images, although this is not excluded here. SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF) are some examples of these feature detection algorithms. All of these create a corresponding feature descriptor for a point by utilizing the pixel texture or pixel values ​​surrounding a single pixel in the image. The feature descriptor determines or defines the characteristics of the detected feature. A major example of creating a feature descriptor for a point by utilizing the pixel values ​​surrounding a single pixel in the image is... Figure 4 The diagram shows that the feature descriptor for pixel p will be created from the pixel values ​​or textures of the remaining twenty-seven pixels surrounding pixel p.

[0087] The use of ORB-based feature descriptors is advantageous in determining the position and orientation of a moving machine 1 because matching ORB-based feature descriptors is substantially robust to changes in imaging conditions such as illumination, blur, rotation, and scaling, and is computationally fast enough for real-time applications.

[0088] Identical features appearing in different images can be matched by comparing their feature descriptors. According to one embodiment, the feature matching process can be implemented using commonly known brute-force search-based problem-solving techniques and algorithmic paradigms, which involve systematically enumerating all possible candidates for a solution and checking whether each candidate satisfies the problem statement.

[0089] Therefore, in the solution disclosed herein, the process related to matching features appearing in different images may include comparing descriptors of all features and selecting the best match to correspond to the same feature in different images. Furthermore, a forward-backward consistency check may be performed, where, for example, it is verified whether the i-th feature descriptor in the first image has the j-th feature descriptor that is the best match in the second image, and vice versa. Finally, only those matches with a sufficiently small feature descriptor distance (i.e., an abstract distance defined for the feature descriptors) are accepted to represent the same feature in different images.

[0090] After obtaining the three-dimensional position coordinates of features in the World Wide Web Control System (WDCS) or the site-specific WCS coordinate system and their corresponding two-dimensional image coordinates in the image (in which these features are visible), along with the pose and position of each image and the feature descriptor, the feature descriptors and their three-dimensional coordinates can be correlated to generate a feature point map, i.e., a navigation map. In the feature point map, each feature is preferably uniquely indexed. In the feature point map disclosed herein for the solution, for example, the following information can be stored for each feature forming the map feature point MFP:

[0091] • Feature index (e.g., an index number starting from 0 or 1);

[0092] • Feature descriptors (such as ORB descriptors or ARUCO codes);

[0093] • Three-dimensional site WCS or world WDCS coordinate system coordinates, which may include an ellipsoid indicating uncertainty in the accuracy of the determined coordinates;

[0094] • Image identifiers that identify images in which the same features can be seen (there are at least one image in which the same features can be seen, and usually at least two images);

[0095] • For images where features are visible, the image coordinates used to locate the features may include ellipses indicating the uncertainty in the accuracy of the image coordinates;

[0096] • The position and orientation of each image in the World WDCS or Site WCS coordinate system.

[0097] • The covariance matrix for the aforementioned features, i.e., accuracy information;

[0098] • Static rating for the aforementioned feature;

[0099] • Semantic classification information, including classifications of man-made or natural objects describing the location of features, such as balconies, construction booths, manholes, dense trees, and gravel.

[0100] • Visible observation directions, such as all directions, north, northeast-west, directions in degrees, or the ground area of ​​the construction site, i.e., the southern parking area.

[0101] A first embodiment of providing map feature points to create a feature point map:

[0102] Figure 5a , Figure 5b and Figure 5cAn embodiment for determining multiple map feature points (MFPs) to create a feature point map is illustrated schematically. Specifically, a first embodiment for determining multiple distinguishable features (DFs) as multiple map feature points (MFPs) based on a site environment (WE) is provided. The multiple map feature points (MFPs) have at least identification information and location data. The identification information includes at least one unique identifier, similar to a feature descriptor, for the map feature point (MFP) to identify it from an image. The location data includes at least one of the following: the three-dimensional position of the map feature point (MFP) in the site 14, i.e., the three-dimensional position in the site coordinate system (WCS), or two-dimensional image coordinates with an image identifier, i.e., information on at least one image from which the map feature point (MFP) is found and the location of the image pixels found therein. In this embodiment, the map feature points (MFPs) for the feature point map are determined using reference feature points (RFPs) already pre-defined in the site 14.

[0103] Figure 5a , Figure 5b and Figure 5c An embodiment includes the step of capturing multiple two-dimensional site environment images (WIMs) using at least one camera (CA, PCA), wherein the at least one camera (CA, PCA) has a defined intrinsic parameter set. Figure 5b Four consecutive site environment images (WIMs) taken from the site environment (WE) are schematically illustrated. These site environment images (WIMs) are preferably taken such that there is significant overlap between the consecutive images, for example, 40% to 80% overlap. For clarity, in Figure 5b In the image, consecutive WIM images are shown separately from each other. The leftmost WIM image can be considered the first WIM image of the construction site environment taken, and the rightmost WIM image can be considered the last WIM image of the construction site environment taken.

[0104] The acquisition of the WIM image can be performed by at least one positioning camera PCA attached to a machine 1 moving around the construction site 14 and / or by at least some other camera CA not attached to any positioning camera PCA of the machine 1. This at least one other camera CA can, for example, be attached to a drone arranged to move within the construction site environment WE. Figure 1 The image clearly illustrates this other camera CA. The focus of the camera CA and PCA is... Figure 5b The diagram is schematically illustrated using black dots marked with the reference symbol CFP.

[0105] Following the step of capturing multiple 2D construction site environment images (WIMs), the next step is to detect at least one of the following from each of the multiple 2D construction site environment images (WIMs):

[0106] a) Distinguishing features (DFs) are used to associate each distinguishable feature DF with the identification information and image coordinates of the corresponding 2D site environment image (WIM), serving as the detected feature point (DFP); or

[0107] b) Reference Feature Points (RFPs), wherein the reference feature points (RFPs) are pre-set in the site 14 and associated with identification information and location data, and are used to additionally associate each reference feature point (RFP) with the image coordinates of the corresponding two-dimensional site environment image (WIM) as a detected reference feature point (DRFP).

[0108] Therefore, the distinguishable feature DF detected from each of the plurality of images may include only one of options a) and b), or both options a) and b). For example, some images may contain only option a), some images may contain only option b), and some images may contain both options a) and b).

[0109] Therefore, the acquired two-dimensional site environment image WIM can include two distinguishable features: the first feature is to be identified as the detected feature point DFP, and the second feature is to be identified as the reference feature point RFP of the detected reference feature point DRFP.

[0110] First, consider the reference feature point RFP, i.e., option b) above. The reference feature point RFP is a reference point already preset in site 14. These can be, for example, the markers discussed above. Figure 5b In this context, these reference feature points (RFPs) correspond to graphic symbols that include circles surrounding a cross. The line of sight from the camera focus (CFP) to the reference feature point (RFP) is schematically shown with corresponding arrows. In the site environment image (WIM), the reference feature points (RFPs) are located at the positions indicated by the asterisk graphic symbols.

[0111] Reference feature points (RFPs) can be identified as detected reference feature points (DRFPs) when associated with their position data and identification information in the 3D world WDCS or site WCS coordinate system and their corresponding image coordinates in the corresponding 2D site environment image (WIM). Therefore, Figure 5b The asterisk graphic symbol in the image can be represented as the detected reference feature point DRFP, and the image coordinates refer to the image pixel coordinates of the asterisk in the corresponding detected reference feature point DRFP in the corresponding site environment image WIM.

[0112] Secondly, consider the distinguishable feature DF of the feature point DFP that will be identified as the detected feature point. Figure 5bThe building 15 is associated with three triangles pointing upwards, and two stones 16 are associated with the triangles pointing upwards, as well as one stone 16 is associated with the triangle pointing downwards. Figure 5b The triangles in the diagram depict distinguishable features (DFs) detected in the site environment image (WIM) and distinct from the preset reference feature points (RFPs) discussed above. The triangles pointing downwards describe distinguishable features (DFs) that cannot or will not match in two or more images, as discussed in more detail below. Figure 5b The diagram illustrates triangles associated with objects considered immovable within site 14, which are preferred from the perspective of creating a feature point map for navigation of machinery 1. The immovable or movable nature of objects within site 14 can be determined using semantic image segmentation known in the art. Alternatively, this can be done manually, for example, by selecting all distinguishable features detected from a vehicle about to leave site 14 and determining that they are located on movable objects. The line of sight from the camera focus (CFP) to the detected triangles is schematically shown with corresponding arrows. In the site environment image (WIM), these triangles are positioned as indicated by the graphic symbol indicated by small black circles.

[0113] When distinguishable features represented by triangles (DFs) are associated with their identification information and their image coordinates in the corresponding 2D site environment image (WIM), they are identified as detected feature points (DFPs). Therefore, in Figure 5b In the image, the graphic symbol of the small black circle can be represented as the detected feature point DFP, and the image coordinates refer to the image pixel coordinates of the small black circle related to the corresponding detected feature point DFP in the corresponding site environment image WIM.

[0114] It is noted here that whenever a new site environment image (WIM) is captured, the process of detecting distinguishable features (DF) and determining detected feature points (DFP) and detected reference feature points (DRFP) is performed, but not necessarily in real time. It is also noted here that some reference feature points (RFP) may initially be incorrectly detected as detected feature points (DFP), but may later be correctly interpreted as detected reference feature points (DRFP).

[0115] When acquiring a site environment image (WIM) and detecting distinguishable features (DFs) within it, at least one image group (GIM) is also created based on the site environment image (WIM) by binding a site environment image (WIPM) that has at least one of the following:

[0116] -Detected feature point DFP, or

[0117] - Detected reference feature points DRFP

[0118] It is matched as the same detected feature point DFP or the same detected reference feature point DRFP in different site environment images WIM.

[0119] When creating at least one image group GIM based on site environment images (WIMs), at least one image group GIM, i.e., at least one image mosaic, is created from the site environment images (WIMs) based on specific conditions discussed below. In this image group GIM, at least two different site environment images (WIMs) taken from different camera positions are grouped into at least the same group, i.e., the image group GIM, based on detected feature points (DFPs) and / or detected reference feature points (DRFPs) that match the same points in these different site environment images (WIMs). At least two site environment images (WIMs) can be bound to an acceptable image group GIM based on the following conditions: in each of the site environment images to be selected into the image group GIM, there are at least five detected feature points (DFPs) or detected reference feature points (DRFPs) that match the same points between each individual site environment image (WIM) and another image in the remainder of the image group; and the entire acceptable image group contains at least three detected reference feature points (DRFPs) that match the same points in all the site environment images (WIMs) of the acceptable image group GIM. More specifically, when binding the next image to an image group, there must be at least five points binding that next image to that image group (detected feature points DFP, detected reference feature points DRFP, or some mixture of the former, such as four DFPs and one DRFP), and these points should be located in a single image already in that image group. Therefore, these five points do not need to be in every other image in the image group. As mentioned earlier, one image with five binding points is sufficient, and more than one is acceptable but not required.

[0120] For example, an image group could contain 100 images taken around a building. Each of these images could contain five points that bind the image to a previous image (except the first image) and subsequent images (except the last image, although the last image can be bound to the first image). Another example could be four images in an image group GIM, where the second image is bound to the first image by five points, the third image is bound to the first image by five points, and the fourth image is bound to the third image by five points and to the second image by five points. Thus, the fourth image is bound to the two images in the image group that were already bound before the fourth image was bound by five points (either detected feature points DFP or detected reference feature points DRFP or a combination of the former).

[0121] When creating an image group as described above, no information external to the site environment image (WIM) is required to make the decision to bind the corresponding site environment image (WIM) to form the image group (GIM). However, if available information about where each image was taken, such as the camera position and orientation when these images were taken, can be used to speed up the binding of these images, because this information can indicate that some images may not even theoretically have the possibility of having a single matching point, for example, if the images were taken from the same location but in opposite directions.

[0122] refer to Figure 5b This means that at least three images, either the leftmost or the rightmost and the second image from the left, can be used to form an acceptable image group. An optimal image group will contain... Figure 5b Images of all four site implementation examples.

[0123] Figure 5c The diagram schematically illustrates an image group GIM formed by five different site environment images (WIMs). These five different site environment images (WIMs) may or may not be consecutive images captured by at least one camera (CA, PCA). However, knowing the order of the images can speed up the grouping process. Figure 5c In order to be clear, and Figure 5b Slightly different, the upward-pointing triangle is now directly represented as the detected feature point DFP that matches the same feature point in at least two site environment images (WIM), and the circle around the cross is now directly represented as the detected reference feature point DRFP that matches the same feature point in at least two site environment images (WIM). Figure 5c Further including apex-pointing triangles, which represent distinguishable features DFs detected but not identified (i.e. not matched) as identical points in two or more WIMs.

[0124] exist Figure 5c In the example, each individual site environment image WIM happens to include at least one detected reference feature point (DRFP). It is also possible that one, two, or even three of the images may not include a single detected reference feature point (DRFP). Figure 5cIn the example, five different site environment images (WIMs) are bound together, i.e., connected to each other and overlapped in at least one direction in the paper plane to form a site environment image group (GIM). This group is acceptable because at least three reference feature points (RFPs) that match as identical points exist in at least two of the images, and each image is bound to some other image that has at least five detected reference feature points (DRFPs) or detected feature points (DFPs) that match as identical points. It should be noted that if any of these images are missing, the group cannot form an acceptable image group because all these images need to be bound together and have at least three reference feature points (RFPs) that match as identical points in at least two of the images.

[0125] Each such group (i.e., each image group GIM containing at least three detected reference feature points (DRFPs) that are matched as identical, and each image bound to at least one other image in the group by at least five points that are matched as identical) is accepted for further analysis to determine map feature points for feature point mapping. If the condition is not met, the image group GIM is rejected for further analysis. Rejection can still be avoided by determining the 3D locations of a sufficient number of detected feature points (DFPs) that are matched as identical, or by any other such points determined from the images (detectable from at least two images, and whose 3D coordinates can be determined as detected reference feature points (DRFPs)). This operation can save time because it is not necessary to take another set of images while the markers are visible. In some cases, the number of images can be much higher than five.

[0126] Return to reference Figure 5bFor example, as described above, the second site environment image WIM from the left and the rightmost site environment image WIM can form an acceptable group of two images because both images involve the same three detected reference feature points (DRFPs), thus containing at least three detected reference feature points (DRFPs) matched as the same in the image group. Furthermore, since both images also involve at least two detected feature points (DFPs) matched as the same, the total number of five DFP / DRFP points matched as the same is satisfied. In this case, there are three different detected reference feature points (DRFPs) shared by the two images. Similarly, for example, three leftmost or three rightmost images can together form corresponding two acceptable image groups. However, for example, two rightmost images cannot form an acceptable image group without any other images, even though they have five additional detected feature points (DFPs) matched as the same, because these two rightmost images only have two identical detected reference feature points (DRFPs) at their respective locations, thus failing to satisfy the condition of containing at least three detected reference feature points (DRFPs) matched as the same in the image group.

[0127] In further analysis, for each accepted image group GIM, the location and orientation of the image group GIM are determined by utilizing the identification information and location data of at least three detected reference feature points (DRFPs) that are matched as identical points contained in the accepted image group GIM, and the identification information and location data of the detected feature points (DFPs) that are matched as identical points are determined, wherein at least one of the following:

[0128] - Detected feature points (DFPs) that are matched with the same point and have definite identification information and location data, or

[0129] - Detected reference feature points (DFRPs) that are matched as identical points and associated with identification information and location data are identified as multiple map feature points (MFPs). In this process, the three-dimensional position of the detected feature point (DFP) of the corresponding image group (GIM) is determined in the site coordinate system (WCS) by using the three-dimensional position of each of at least three detected reference feature points that are matched as identical points.

[0130] In other words, only those detected feature points (DFP) and detected reference feature points (DRFP) in the accepted image group GIM can be selected as map feature points (MFP) for navigation maps. The detected feature points (DFP) and detected reference feature points (DRFP) are matched as the same points based on their determined identification information determined in the step of detecting distinguishable features (DF) from each of the multiple 2D site environment images (WIM). Figure 5d It schematically shows that in Figure 5cExamples of accepted map feature points (MFPs) selected from the detected feature point (DFP) and the detected reference feature point (DRFP) are shown.

[0131] According to the embodiment of determining map feature points (MFPs) disclosed herein, the steps of determining the location and orientation of each accepted image group (GIM) and determining identification information and location data for detected feature points (DFPs) matched as the same further include determining a static rating of the detected feature points (DFPs) matched as the same by utilizing the identification information and location data of at least three detected reference feature points (DFRPs) included in the accepted image group, wherein at least one of the following:

[0132] - DFPs that are matched as the same detected feature points, having definite identification information, location data, and static rating, or

[0133] - Detected reference feature points (DFRPs) associated with identification information and location data.

[0134] It was identified as multiple map feature points (MFP).

[0135] Static rating provides a measure of the accuracy of the location of detected feature points (DFPs) to be applied as map feature points (MFPs). This accuracy information is determined by utilizing the identification information and location data of at least three detected reference feature points (DRFPs). The accuracy of each detected feature point (DFP) can be significantly affected by factors such as the distance between the detected DFP and the camera during image capture; for example, the farther the detected DFP is from the camera, the less accurate the information obtained may be. Most of the information related to the static rating of the detected feature point (DFP) is determined during the bundle adjustment discussed above and is represented by the corresponding covariance matrix for the 3D location of the respective detected feature point (DFP). Therefore, the static rating can be an ellipsoid indicating the uncertainty of the accuracy of the determined 3D coordinates.

[0136] The static rating of the detected reference feature points (DFRPs) is considered to be 100% or very close to 100% because the detected DFRPs correspond to preset markers in site 14, and their location data is known in advance. In the best-case scenario, the static rating of the detected feature points (DFPs) can reach 100% through very accurate calculations during bundle adjustment; however, in typical cases, the accuracy of the detected feature points (DFPs) is slightly lower.

[0137] Semantic image segmentation can be used to consider whether an object in site 14 is immovable or movable, and this will affect the static rating of the detected feature points (DFPs) on the object. To some extent, semantic image segmentation can be performed manually, for example, by selecting all distinguishable features or detected feature points (DFPs) detected from a car about to leave site 14 and determining that they are located on movable objects. This would ultimately result in these points being considered to have the lowest static rating and thus not being selected as map feature points (MFPs). Nevertheless, these points can be used when the images are bound to an image group (GIM). In this case, it should be remembered that map feature points (MFPs) will be missing in the area when the car leaves. A discussion related to rating can be found later in this specification.

[0138] A second embodiment of providing map feature points to create a feature point map:

[0139] Figure 6 A second embodiment for determining multiple map feature points (MFPs) to create a feature point map is illustrated schematically. Specifically, this embodiment is for determining multiple distinguishable features (DFs) as multiple map feature points (MFPs) from a site environment (WE). The multiple map feature points (MFPs) have at least identification information and location data. The identification information includes at least one unique identifier, similar to a feature descriptor, for identifying the map feature point (MFP) from an image. The location data includes at least one of the following: the three-dimensional location of the map feature point (MFP) in the site (14), or two-dimensional image coordinates with an image identifier. In this embodiment, the three-dimensional location of the map feature point (MFP) for the feature point map is determined using a satellite positioning system (GNSS). At least one possible sensor is applied to a camera (CA) or camera (PCA) to determine the camera's position. If the orientation information of the camera (CA) or camera (PCA) is known, it can be used in conjunction with location / positioning information.

[0140] This embodiment includes the step of capturing multiple two-dimensional site environment images (WIMs) using at least one camera (CA, PCA), wherein the at least one camera (CA, PCA) has a defined set of intrinsic parameters, a defined position in the site coordinate system (WCS), and a possibly defined orientation. As discussed earlier, the site environment images (WIMs) are preferably captured such that there is significant overlap between consecutive images, for example, 40% to 80%. The captured site environment images (WIMs) can be compared with... Figure 5b Similar to that shown, however, the difference lies in that the reference feature points RFP, which may be pre-defined in construction site 14 and visible in the captured construction site environment image WIM, can be considered as... Figure 5bThe reference feature point (RFP) is a distinguishable feature similar to the detected feature point (DFP). In other words, in this embodiment, the reference feature point (RFP) does not need to be used as a reference feature point with a predetermined three-dimensional location, but can be used if necessary.

[0141] Image WIM acquisition can be performed by at least one positioning camera PCA attached to a machine 1 moving around the site 14 and / or by at least some other camera CA that is not attached to any positioning camera PCA of the machine 1. Furthermore, this at least one other camera CA can, for example, be attached to a drone arranged to move within the site environment WE, provided that the position of the camera CA can be determined at least with respect to each image WIM. If determined, orientation information of the camera CA can also be used.

[0142] After capturing multiple 2D construction site environment images (WIMs), the next step is to detect distinguishable features (DFs) from each of the multiple 2D construction site environment images (WIMs) and determine the distinguishable feature (DFP) as the detected feature point (DFP) using the image coordinates of the corresponding 2D construction site environment images (WIMs).

[0143] Therefore, in this paper, the acquired two-dimensional construction site environment image WIM includes only one distinguishable feature, namely, it is considered to correspond to Figure 5b and 5c The detected feature point DFP is characterized by a triangle pointing upwards or downwards. Furthermore, as mentioned above, if detected, possible preset reference feature points RFP in site 14 (visible in the site environment image WIM) are also considered as detected feature points DFP, and similarly, location information from them is not required, but can be used if needed. Therefore, if the graphic symbols associated with the reference feature point RFP and the detected reference feature point DRFP therein are omitted or considered to be replaced by a triangular graphic symbol associated with the detected feature point DFP, then... Figure 5b and Figure 5c The disclosures herein apply. When distinguishable features represented by triangles (DF) are associated with their image coordinates (i.e., their image pixel coordinates) in the corresponding two-dimensional site environment image (WIM), these distinguishable features are identified as detected feature points (DFP).

[0144] It is worth noting that as soon as a new site environment image WIM is acquired, the process of detecting distinguishable features DF and detected feature points DFP is performed, but it is not necessarily in real time.

[0145] When acquiring a site environment image (WIM) and detecting distinguishable features (DFs) within it, at least one image group (GIM) is also created from the site environment images (WIMs) by binding site environment images (WIMs) that have detected feature points (DFPs) that are matched as the same in different site environment images (WIMs). The process for creating at least one image group is similar to the process disclosed above; however, the difference is that only detected feature points (DFPs) are considered here.

[0146] In addition, in order to determine the map feature points (MFP) for the feature point map of site 14, by utilizing the position and orientation of at least one camera (CA, PCA) in the site coordinate system (WCS) and each corresponding two-dimensional site environment image (WIM), identification information and location data for the detected feature points (DFP) that are matched as the same are determined for each image group (GIM), wherein the detected feature points (DFP) that are matched as the same and have determined identification information and location data are determined as multiple map feature points (MFP).

[0147] In other words, only those detected feature points (DFPs) in the image group GIM will be selected as map feature points (MFPs) for the feature point map. These detected feature points (DFPs) are matched as identical points based on the specific identification information that has been determined in each step of detecting distinguishable features (DFs) from multiple 2D site environment images (WIMs).

[0148] According to the embodiments for determining map feature points (MFPs) disclosed herein, the step of determining the identification information and location data for each image group (GIM) to be matched as the same detected feature point (DFP) further includes determining a static rating for the DFPs to be matched as the same by utilizing the position and orientation of at least one camera (CA) and PCA in the site coordinate system (WCS) with each corresponding two-dimensional site environment image (WIM), wherein the DFPs to be matched as the same detected feature point with determined identification information, location data and static rating are determined as a plurality of map feature points (MFPs).

[0149] The above discussion related to static rating also applies here, except that here, static rating is determined by utilizing the position and possible orientation of at least one camera CA and PCA in the site coordinate system WCS with each corresponding 2D site environment image WIM, rather than the identification information and position data of at least three detected reference feature points DRFP. In other words, when the image is captured, the position and orientation data of the camera CA and PCA of the captured image, along with their accuracy information—i.e., a determined accuracy estimate—can be used at the precise moment the image is captured. Furthermore, static rating may include the accuracy information of the detected feature points DFP obtained from image position / orientation measurements and bundle adjustment calculations.

[0150] Differences in the static rating of map feature points (MFPs) arise from factors such as the location of the MFP, the distance between the camera's CA and PCA at the time the image was captured, and the type of distinguishable feature (DF) that makes the MFP a map feature point. If an MFP is located on a thin tree or branch, semantic classification may assign it a lower rating compared to a thick tree or branch detected under approximately the same conditions. Conversely, under the same conditions, an MFP located at the corner of a building window may receive a higher rating. Nevertheless, the superiority of the static rating for MFPs located at building window corners, thin trees, thin branches, thick trees, or thick branches varies considerably depending on factors such as differences in the distance from the camera's CA and PCA to the objects and / or the accuracy of the camera's CA and PCA at the time these images were captured, different weather or lighting conditions at the time each image was captured, or different weather or lighting conditions for each object—i.e., whether each object was imaged in shadow, sunshine, fog, or rain. In short, many variables influence the static rating. Information related to static ratings and the static ratings themselves can be included in the identification information of each map feature point (MFP).

[0151] Rating is useful when a large number of map feature points (MFPs) are available because the location and orientation of machinery can be determined using map feature point MFPs with higher ratings, i.e., by discarding map feature point MFPs with lower ratings at the current location, at least temporarily. For example, a rating threshold can be selected, where map feature point MFPs with excessively low values ​​are not used when determining the location and orientation of machinery 1. Another approach is to select a number of the highest-rated map feature point MFPs located in the area pointed to by the positioning camera PCA, relative to the most recently determined location and orientation. Methods that consider previously determined locations and orientations of the positioning camera PCA can further utilize dynamic rating, discussed later.

[0152] According to one embodiment, an overall rating is further determined for the detected feature points (DFPs), the overall rating including at least one of a static rating or a dynamic rating, i.e., a static rating and / or a dynamic rating, and for each of the plurality of determined map feature points (MFPs), a dynamic rating is determined individually with respect to each positioning camera (PCA), taking into account the previously determined position and orientation or the orientation of each of at least one positioning camera (PCA). The orientation or orientation of each of at least one positioning camera (PCA) refers here to the viewing sector of each positioning camera (PCA). If many map feature points (MFPs) are not even close to the visual area being viewed by the positioning camera (PCA), these map feature points (MFPs) will receive the lowest dynamic rating for that positioning camera (PCA), which can be, for example, 0, i.e., zero or empty. The overall rating can be the sum of the static and dynamic ratings, or the overall rating can be the product of the static and dynamic ratings. Thus, for example, a map feature point located behind a positioning camera (PCA) will receive an overall rating of zero, meaning that determining the position and orientation of the machine will not attempt to find such a map feature point (MFP) from an image taken at that time by the corresponding positioning camera (PCA). On the other hand, if there is another positioning camera PCA on the other side of machine 1, whose orientation or location differs from the previously mentioned "one positioning camera PCA" by approximately 180 degrees, then this other positioning camera PCA may assign a dynamic rating of 1, or the highest or fullest, to the corresponding map feature point MPF. Similarly, there is a possibility that the map feature point MPF in question may be located between the two positioning camera PCAs, and therefore the dynamic rating of both cameras would be 0, i.e., zero or empty.

[0153] According to this embodiment, as discussed above, a dynamic rating is determined for each of a plurality of determined map feature points (MFPs) by individually considering the previously determined position and orientation of each of at least one positioning camera (PCA). Here, additionally or alternatively, different weights, i.e., dynamic ratings, can be applied to each map feature point (MFP) based on the realized distance between the respective positioning camera (PCA) and the detected feature point (DFP) corresponding to the map feature point (MFP) in question.

[0154] Then, the overall rating is a static or dynamic rating, or a combination or weighted combination of static and dynamic ratings, determined by the map feature points (MFP) and the positioning camera (PCA) for each location camera in question.

[0155] According to one embodiment, the overall rating is at least one of a static rating or a dynamic rating, and the overall rating is at least two-tiered.

[0156] According to this embodiment, the overall rating is a static rating, a dynamic rating, or a combination of static and dynamic ratings. Furthermore, the overall rating is at least two-tiered, i.e., includes at least two distinct levels, whereby each map feature point MFP associated with a corresponding detected feature point DFP, and its accuracy or other data related to its usefulness at each time point, can be classified into different rating categories or ratings based on, for example, the accuracy of the detected feature point DFP corresponding to the map feature point MFP in question, other identification information, and the position and orientation (i.e., its viewing sector) of each map feature point MFP relative to each positioning camera PCA. The map feature point corresponding to the detected reference feature point DRFP typically has the highest accuracy due to preset markings at known locations in site 14, and is therefore typically classified with the highest level.

[0157] According to one embodiment, if the number of multiple map feature points exceeds a threshold, at least the map feature point MFP with the lowest overall rating is discarded, wherein the threshold is determined manually and / or automatically.

[0158] According to this embodiment, when the number of map feature points (MFPs) applicable to the feature point map is unnecessarily high, map feature point MFPs with low accuracy can be discarded, as discussed earlier regarding the usefulness of the rating. This reduces the computational power required to determine the location and orientation of machine 1 and / or increases the speed of determining the location and orientation of machine 1. The threshold can be determined, for example, based on historical data regarding the number of reference point DFPs detected in the site 14 under discussion and other relevant data, and can be updated based on data received during the currently ongoing work phase.

[0159] A third embodiment of providing map feature points to create a feature point map:

[0160] A third embodiment for determining multiple map feature points (MFPs) to create a feature point map is disclosed herein, namely, a third embodiment for determining multiple distinguishable features (DFs) from a site environment (WE) as multiple map feature points (MFPs), wherein the multiple map feature points (MFPs) have at least identification information and location data, wherein the identification information includes at least one unique identifier, which is similar to a feature descriptor, for the map feature points (MFPs), and the location data includes at least the three-dimensional location of the map feature points (MFPs) in the site (14).

[0161] According to this third embodiment, the map feature points (MFP) used to create the feature point map are determined at least in part based on data retrieved from an earthwork information model, wherein the earthwork information model is based on at least one of the following: Geospatial Information System (GIS), Building Information Modeling (BIM), Infrastructure Building Information Modeling (I-BIM), Civil Information Modeling (CIM), or a smart city platform. The earthwork information model includes a classification system that can define the meaning and characteristics of different types of infrastructure and models for site 14 and their different substructures (such as different foundation layers, infill layers, surface layers, etc.). In the earthwork information model, the planned final structure may have been divided into many substructures to be completed to achieve the planned final structure. Each work phase may further include specific work instructions, or even be in the form of many specific individual operations to be performed by machinery 1 to complete the work phase in question.

[0162] According to this embodiment, the earthwork information model may include, for example, at least some distinguishable features (DFs) based on the geometric information of some structures in the model, the three-dimensional locations of which are known in the model and can be retrieved from the model, to serve as the basis for a new map feature point (MFP). The basis of the new map feature point (MFP) may include information from identification information and location data, but the identification information may not include feature descriptors required for the feature to become a map feature point (MFP). Feature descriptors can be generated by capturing at least one image containing the feature area and its surrounding environment. Alternatively, it may be possible to identify which locations are identifiable in the earthwork information model (i.e., the earthwork information model) from the generated map feature point (MFP) or from distinguishable feature DFs determined based on multiple two-dimensional site environment images (WIMs) capturing such features, and generate location data for these features based on the earthwork information model, and generate feature descriptors based on at least one image where the feature can be found as a distinguishable feature. If the feature descriptor is taken from a distinguishable feature found only in one image, the generated map feature point (MFP) will contain only one feature descriptor; however, there may be more than one feature descriptor, each associated with a corresponding image in which the feature exists.

[0163] Alternatively, the step of determining multiple distinguishable features (DFs) from the site environment (WE) as multiple map feature points (MFPs) further includes determining additional map feature points (MFPs) based at least in part on data retrieved from an earthwork information model, wherein the earthwork information model is based on at least one of: a geospatial information system (GIS), building information modeling (BIM), infrastructure building information modeling (I-BIM), civil information modeling (CIM), or a smart city platform.

[0164] Alternatively, based on the first embodiment of the map feature point MFP for creating feature point maps disclosed above, or the second embodiment of the map feature point MFP for creating feature point maps disclosed above, and further based at least in part on data retrieved from an earthwork information model, a plurality of map feature point MFPs are determined, wherein the earthwork information model is based on at least one of the following: a geospatial information system (GIS), building information modeling (BIM), infrastructure building information modeling (I-BIM), civil information modeling (CIM), or a smart city platform.

[0165] Figure 8 Some earthwork engineering information models, namely earthwork information models, are disclosed schematically.

[0166] A fourth embodiment of providing map feature points to create a feature point map:

[0167] A fourth embodiment for determining multiple map feature points (MFPs) to create a feature point map is disclosed herein, namely, a fourth embodiment for determining multiple distinguishable features (DFs) from a site environment (WE) as multiple map feature points (MFPs), wherein the multiple map feature points (MFPs) have at least identification information and location data, wherein the identification information includes at least one unique identifier, which is similar to a feature descriptor, for the map feature points (MFPs), and the location data includes at least the three-dimensional location of the map feature points (MFPs) in the site (14).

[0168] According to the fourth embodiment, based on the method disclosed in at least two of the first, second, or third embodiments above, the map feature points MFP for creating a feature point map are determined as a combination set of map feature points MFPs to provide map feature points for creating a feature point map.

[0169] Return to reference Figure 2a and Figure 2b And related descriptions, Figure 7a , 7b and Figure 7c Combination Figure 2a and Figure 2b An embodiment for determining the position and orientation of machine 1 in site 14 is illustrated schematically. When a pre-created feature point map including the aforementioned map feature points MFP is available, the position and orientation of machine 1 moving in site 14 can be determined.

[0170] To achieve this, the method includes: during operation of machine 1, substantially continuously capturing two-dimensional positioning images (PIMs) by each of at least one positioning camera (PCA) at very short time intervals (e.g., between 10 and 100 milliseconds), with an average time interval of approximately 50 milliseconds in each positioning camera (PCA). The time interval for capturing the positioning images (PIMs) can be extended if machine 1 is stopped at site 14 without performing any operational actions to achieve a work phase.

[0171] While capturing 2D positioning images (PIMs), distinguishable features (DFs) are detected from the PIMs as navigation feature points (NFPs), and the 2D image coordinates (i.e., image pixel coordinates) of each navigation feature point (NFP) in each corresponding PIM are determined. Each navigation feature point (NFP) has at least identification information. Therefore, the content of these 2D positioning images (PIMs) is somewhat similar to, for example... Figure 5c The content of the WIM image of the construction site environment shown is as follows: Figure 7b It is visible in the text.

[0172] While capturing a two-dimensional positioning image (PIM) and analyzing its content to detect distinguishable features (DF) from the positioning image PIM as navigation feature points (NFPs) and determining the two-dimensional image coordinates of each navigation feature point (NFP) in the positioning image, the navigation feature point (NFP) is matched with multiple map feature points (MFPs) in a pre-created feature point map. Thus, the identification information of the navigation feature point is matched with the identification information of at least one of the map feature points (MFPs) or the detected mismatched feature points (DFM-NM).

[0173] In response to the matching of navigation feature points (NFP) and map feature points (MFP), the location and orientation of machine 1 on the construction site can be determined based on at least the following:

[0174] -Location data of at least one of the matched map feature points (MFP) or the detected mismatched feature points (DFP-NM).

[0175] - The corresponding two-dimensional image coordinates of the matched navigation feature points (NFP), and

[0176] - A defined set of intrinsic parameters and a defined position and orientation in the mechanical coordinate system (MCS) of each corresponding positioning camera (PCA).

[0177] The position and orientation of machine 1 can be determined in at least two different ways, and in both ways, additional robustness and / or accuracy can be provided by utilizing sensors and / or sensor arrays, such as inertial measurement units (IMUs) located in machine 1.

[0178] According to the first embodiment or method described herein, a PnP (Perspective n-Point Problem) solution can be applied, which uses the three-dimensional information of map feature points MFPs that match navigation feature points NFPs. In this solution, the position and orientation of at least one positioning camera PCA in site 14 are first determined based on the position data of each matched map feature point MFP and the two-dimensional image coordinates of each corresponding matched navigation feature point NFP. Therefore, the input to the PnP problem is the three-dimensional position data of the matched map feature points MFPs and the two-dimensional image coordinates of the corresponding matched navigation feature points NFPs, and the output of the PnP problem is the position and orientation of at least one positioning camera PCA in site 14. Based on the determined intrinsic parameter set and the determined position and orientation of each corresponding positioning camera PCA in the machine coordinate system MCS, the determined position and orientation of at least one positioning camera PCA in site 14 can be converted into the position and orientation of machine 1 in site 14. The PnP problem is usually discussed in more detail in publications such as Fischler and Bolles' "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography" (Communications of the ACM, June 1981, Vol. 24, No. 6).

[0179] The positioning results determined above can be improved through statistical estimation, for example, by solving a nonlinear optimization problem of variables. This nonlinear optimization problem, assuming a normal distribution of measurement errors, stems from well-known Bayesian statistics, which assumes that measurement errors are normally distributed. The basic idea here is to compare feature measurement results, i.e., to compare the image coordinates of the navigation feature point NFP from the camera and possible inertial measurements from the IMU and / or possible other measurements from other sensors or sensor groups with simulated / artificial measurements calculated by the camera projection model and possible IMU kinematic models and / or possible models of the other sensors or sensor groups, and to assume some values ​​for the state variables to be solved (the current state estimate). Based on the comparison, the current state estimate is changed to reduce the comparison results. This adjustment process may be repeated until convergence, i.e., no further significant improvement is obtained.

[0180] Information about the 3D positions of features that match map feature points in the feature point map can be incorporated into the optimization problem as prior information for the 3D positions of the features, which is now included as a state variable to be solved. In practice, this means that the function to be minimized in the optimization problem includes terms that compare the current estimate of the 3D position of the feature matched with the map feature point MFP with the position values ​​stored in the feature point map. Now, when adjusting the state estimates during optimization, these terms cause the solution to ensure that the 3D feature positions calculated from camera measurements do not deviate from the 3D positions stored in the feature point map, at least not more than allowed by the uncertainty information of the 3D positions of the map feature points, thus preventing drift from accumulating in the determined localization results.

[0181] According to the second embodiment or method described herein, the position and orientation of machine 1 are determined by a single image. Such an image, where the position and orientation are accurately determined, has known intrinsic parameters of the camera CA and PCA used to capture the image, and includes at least five detected points with feature descriptors, such as ORB descriptors or ARUCO codes, to enable feature detection via the positioning camera PCA. Therefore, in the second embodiment or method, in addition to map feature points (MFPs), detected mismatched feature points (DFPs) can also be used if they do not match in at least two images, and at least contain a feature descriptor, an image identifier identifying the image where the feature can be found, and the two-dimensional image coordinates of the feature in the image. The image identifier and two-dimensional image coordinates can exist in a field that determines the two-dimensional image coordinates via the image identifier. These detected feature points (DFPs) that do not match in two or more images can be very useful, for example, in the edge areas of a construction site or the edge areas of a work area where machine positioning is believed to occur.

[0182] According to the second embodiment or method described herein, at least five navigation feature points (NFPs) are detected in at least one positioning camera (PCA). These NFPs match at least one of the following: map feature points (MFPs) or detected mismatched feature points (DFP-NMs), and these points are located in a single image whose position and orientation are known, i.e., the camera's CA, PCA position and orientation are known when the camera CA, PCA captured the image. Such an image is, for example, an image of an accepted image set (GIM), because when the position and orientation of each in the accepted image set (GIM) are determined, the position and orientation of each image in the GIM are also determined. In this embodiment, the position data of the map feature points (MFPs) and the detected mismatched feature points (DFP-NMs) includes fields for determining two-dimensional image coordinates using image identifiers. In each map feature point (MFP), the number of these position data fields for determining two-dimensional image coordinates using image identifiers is at least one, and can be as many as the number of images with the corresponding detected feature (whose position and orientation are determined). In the detected feature points (DFP-NMs) that are not matched as identical points, the number of these fields for determining two-dimensional image coordinates using image identifiers is only one. For example, in Figure 5b , Figure 5c and Figure 7c The triangle with its tip pointing downwards is shown in the image, and the reference symbol DFP-NM is used to indicate the detected feature points that were not matched as the same.

[0183] In this embodiment or method, the identical image identifier in at least one field of the location data of map feature points MFP or detected mismatched feature points DFP-NM verifies that at least five of these points, representing at least one of the map feature points MFP or detected mismatched feature points DFP-NM, are located in the same image, and the image location and orientation data of the image can be determined by the image identifier, i.e., the location and orientation data of the camera when the image was captured. The two-dimensional image coordinates associated with the corresponding image identifier fields of at least five of the map feature points MFP or detected mismatched feature points DFP-NM indicate where the corresponding map feature point MFP or detected mismatched feature point DFP-NM was found in the image pixels of the image. It should be noted that in this second solution or method, if the two-dimensional positioning image PIM can detect at least five of them, and the location data of these points verifies that these points have the same image identifier in the location data field, and the image identifier indicates the location and orientation of the image, then the location and orientation of machine 1 can even be determined by using only the detected mismatched feature points DFP-NM, for example, by using... Figure 5c The leftmost or rightmost construction site environment image in WIM.

[0184] Now, when at least five map feature points (MFPs) or detected mismatched feature points (DFP-NMs) are found from a single positioning image (PIM), the position and orientation of the positioning camera (PCA) can be determined by using these at least five navigation feature points (NFPs) that match the map feature points (MFPs) and / or the detected mismatched feature points (DFP-NMs) in the single image. This is done, for example, when determining the position and orientation of each accepted image group (GIM), or for example, when capturing images with a camera (CA, PCA) whose position and orientation are determined at the time of image capture.

[0185] In this second embodiment or approach, accuracy improvements can be performed through optimization, as in the first embodiment or approach.

[0186] This solution provides robust and accurate determination of the position and orientation of machinery within a construction site environment. The solution does not accumulate errors in determining the position and orientation of machinery. It creates multiple unique map feature points based on several distinguishable features, which are applied by the earthmoving machinery as it moves and operates within the site. The map feature points include identification information and location data. The identification information includes at least a unique identifier such as a feature descriptor, and the location data includes at least its three-dimensional position within the construction site environment, and / or includes a location data field that includes two-dimensional image coordinates with an image identifier, thereby identifying which image(s) contain the detected map feature point, and wherein the map feature point MFP is located within this image(s).

[0187] Referring to the above discussion concerning the application of semantic information, the robustness of creating feature point maps can be further improved by utilizing semantic information obtained from distinguishable features. Semantic information can, for example, include information about the persistence of objects associated with at least one distinguishable feature (DF). Objects that may completely disappear from the site after a period of time or may at least change their position within the site are preferably not used as fixed points for map feature points in a feature point map intended for long-term use. Therefore, semantic information can be applied, for example, to intentionally assign map feature points to objects that are robustly present in the site environment.

[0188] By calibrating the camera intended for the application in question, the accuracy of feature point creation and the accuracy of determining the position and orientation of earthmoving machinery on the site can also be improved. The camera intended for detecting distinguishable features applicable to map feature points can be calibrated at the beginning of the process for creating a feature point map. Alternatively, the camera intended for detecting distinguishable features applicable to navigation feature points can be calibrated in real time during navigation, while the earthmoving machinery is traveling or operating on the site. When calibrating the camera, its intrinsic parameters are estimated. The camera's intrinsic parameters relate to its internal characteristics, such as its image center, focal length, skewness, and lens distortion.

[0189] To increase robustness in determining the position and orientation of machinery in real time during its travel and operation (e.g., against changes in camera model parameters due to shocks and vibrations), intrinsic camera parameters can be included as variables to be optimized in a nonlinear optimization problem constructed from data collected during navigation. This nonlinear optimization problem is similar to that described above, except that the set of intrinsic parameters for the positioning camera PCA is now optimized / improved along with other parameters. This requires that the camera parameters be observable from the data (i.e., feature measurements obtained from different camera positions), and therefore, data that makes the camera intrinsic parameters observable should be selected. Given a specific dataset collected during navigation, measurements for observability can be defined based on the uncertainty of the estimated parameters (such as the covariance matrix). Here, the specific dataset can be a specific segment of the camera path containing feature measurements taken as the camera moves through that path. If the observability measurements are good enough, the parameters are updated. Furthermore, multiple datasets with good observability of the camera intrinsic parameters can be included in this estimation. Furthermore, changes in intrinsic camera parameters can be detected by comparing two different sets of estimates for these parameters—for example, estimates based on a recent dataset with good observability versus estimates based on numerous datasets collected over a longer period: such differences indicate changes in the intrinsic camera parameters. This information can be used to determine when the parameters need to be re-estimated. This estimation process is performed in parallel with the localization process. After successfully solving for the parameters in this parallel process, the intrinsic camera parameters used in the localization process are replaced with the new parameters.

[0190] According to one embodiment, the step of determining the position and orientation of earthmoving machinery 1 in site 14 is further based on tracking navigation feature points (NFPs) between successive positioning images (PIMs), wherein the navigation feature points (NFPs) tracked between successive positioning images (PIMs) indicate changes in the position and orientation of the positioning camera (PCA).

[0191] This embodiment provides a visual odometry program that can be applied to determine the position and orientation of machinery at site 14 in the short term when, for some reason, the navigation feature point (NFP) detected from the positioning image (PIM) cannot match the map feature point (MFP) in the feature point map. Reasons for using visual odometry might include fog, snow, rain, extremely clear weather, frost, or dust storms. In this way, the determination of the machinery's position and orientation is even more robust and can be managed for a short period when there are not enough navigation feature points (NFP) to match the map feature point (MFP).

[0192] When matching of navigation feature points (NFPs) with map feature points (MFPs) in the feature point map cannot be performed within a short period, a process for simultaneous localization and mapping can be initiated. This involves establishing a temporary feature point map based on the solved 3D position of features tracked in the captured localization image, and locating the machine 1 within this temporary feature point map. In the temporary feature point map, the detected features are used as map feature points.

[0193] Figure 9 An embodiment related to the visual odometry process is illustrated schematically. Figure 9 The dashed line schematically shows machine 1 at a location within site 14 during the capture of the previous positioning image (PIM), and the solid line schematically shows the location of the machine during the capture of the PIM. Figure 9 The machine 1 is located at this position on site 14 during the positioning image PIM shown. Features detected by the positioning camera PCA in site 14 are represented by a solid cross graphic symbol. In the positioning image PIM, these detected features are positioned at locations indicated by a diamond graphic symbol. The dotted lines in the positioning image PIM represent the transfer of the position of the detected features in the positioning image between the previous positioning image and the current positioning image, reflecting the change in the position and orientation of the positioning camera PCA that captured the positioning image PIM.

[0194] During simultaneous localization and mapping, features detected in the latest localization image are also attempted to be matched with the temporary feature point map, not just the actual feature point map. Thus, assuming the created temporary feature point map has good accuracy, good localization accuracy can be maintained even when unsuccessful feature matching with the actual feature map persists for extended periods.

[0195] To establish a temporary feature point map with good accuracy, it can be updated after successful loop detection and loop closure. Loop detection means identifying previously visited areas of site 14—areas visible in one or more previous location images (PIMs)—by comparing the latest location image with those taken at a previous time. Feature values, such as so-called bag-of-words, can be calculated from the location images used for this comparison. Once a loop is detected, if positioning is performed using only odometry, the cumulative drift during the exploration phase, i.e., positioning error, can be corrected using the loop closure function, which takes into account the constraints of the exploration path that forms the loop, i.e., loop constraints, and re-optimizes the paths and local maps established during exploration using these constraints. This process runs in parallel with the real-time positioning process. After successful re-optimization, the temporary feature point map can be updated with new map points.

[0196] According to one embodiment, the location and orientation of earthmoving machinery 1 are further determined in an earthmoving information model, wherein the earthmoving information is based on at least one of: Geospatial Information System (GIS), Building Information Modeling (BIM), Infrastructure Building Information Modeling (I-BIM), Civil Information Modeling (CIM), and a smart city platform. According to this embodiment, the earthmoving information model for site 14 can be updated substantially in real time to include information about the location and orientation of machinery 1 within site 14. For example, this information can be used for further possible measurements to detect potential intersections with safety boundaries set around other machinery or personnel operating within site 14, and to control machinery 1 to stop if necessary.

[0197] According to one embodiment, the tools of earthmoving machinery 1 are controlled to create a structure described in an earthmoving information model. According to this embodiment, machinery 1 can receive, for example, specific work instructions from the earthmoving information model, or even instructions in the form of specific control actions to be performed by control unit 11, in order to complete the structure of the anticipated plan described in the earthmoving information model.

[0198] According to one embodiment, as-built data is generated to accompany the earthwork information model. According to this embodiment, the earthwork information model can be supplemented with as-built data describing the progress of site 14. As-built data regarding the progress of site 14 can be generated by or within the earthmoving machinery 1 while it is moving through site 14, or while it is performing work tasks to complete the planned structure of site 14. Therefore, machinery 1 can collect as-built data not only regarding the progress of site 14 in response to work tasks performed by machinery 1, but also regarding the progress of site 14 in response to work tasks performed by other machinery within site 14.

[0199] According to one embodiment, as-built data is transmitted to a site information management system or to at least one of the machines operating at site 14. The as-built data may be transmitted in response to the completion of the planned structure of site 14, in response to the completion of a substructure forming part of the planned structure, or in response to the completion of a specific individual operation to be performed to implement the substructure.

[0200] Those skilled in the art will understand that, with advancements in technology, the concepts of this invention can be implemented in different ways. The invention and its embodiments are not limited to the examples described above, but can be varied within the scope of the claims.

Claims

1. A method for determining the position and orientation of earthmoving machinery in a site coordinate system, the method comprising: The feature point map is generated using the following method: Multiple distinguishable features are determined from the construction site environment as multiple map feature points by taking multiple two-dimensional images of the construction site environment by at least one camera with a defined set of intrinsic parameters, wherein the set of intrinsic parameters defines the formation of each image pixel from the real-world view. Detect at least one of the following from each of the plurality of two-dimensional construction site environment images: Distinguishing features are used to associate each distinguishable feature with the corresponding identifier information and image coordinates of a two-dimensional construction site environment image, as detected feature points; or Distinguishing features are used as reference feature points, wherein the reference feature points are preset in the construction site and associated with the identification information and location data in the construction site coordinate system, and are used to additionally associate each reference feature point with the image coordinates of the corresponding two-dimensional construction site environment image as a detected reference feature point; Create at least one set of images using the following methods: The site environment images are bound together by having at least one of the following: detected feature points that are matched as identical points in the site environment images, or detected reference feature points that are matched as identical points in the site environment images to be bound. Each set of images contains at least two site environment images and at least three detected reference feature points that are matched as identical points; and By utilizing the identification information and location data of the at least three detected reference feature points, the position and orientation of each accepted image group are determined, and the identification information and location data of detected feature points matched as identical points are determined; thereby The detected feature points that are matched with the same point and the detected reference feature points that are matched with the same point are determined as map feature points for the feature point map; Wherein, the plurality of map feature points in the feature point map have at least identification information and location data, wherein: The identification information includes at least a unique identifier that identifies the distinguishable feature; and The location data includes the three-dimensional location in the site coordinate system; At least one positioning camera is installed on the earthmoving machinery, and the at least one positioning camera has a defined set of intrinsic parameters and a defined position and orientation in the machine coordinate system; Provide the feature point map to the earthmoving machinery; Two-dimensional positioning images are captured using at least one positioning camera; Distinguishing features are detected from the two-dimensional positioning image as navigation feature points, and two-dimensional image coordinates and identification information are determined for each navigation feature point; and The identification information of the navigation feature points is matched with the identification information of map feature points on the feature point map; wherein... The position and orientation of the earthmoving machinery in the site coordinate system are determined based on at least the following: Location data of matched map feature points; The corresponding two-dimensional image coordinates of the matched navigation feature points; and The defined intrinsic parameter set of each of the at least one positioning cameras and the defined position and orientation of each corresponding positioning camera in the mechanical coordinate system.

2. The method as described in claim 1, wherein, The step of identifying multiple distinguishable features from the construction site environment as multiple map feature points further includes: Multiple two-dimensional images of the construction site environment are captured by at least one camera, wherein the at least one camera has: A defined set of intrinsic parameters, and The determined position in the site coordinate system; Distinguishing features are detected from each of the plurality of two-dimensional construction site environment images, and the distinguishing features with the corresponding image coordinates of the two-dimensional construction site environment image are determined as the detected feature points. Create at least one set of images containing detected feature points that are matched as identical points in the site environment images; and The identification information and location data for detected feature points that are matched as identical are determined for each set of images by utilizing the position of the at least one camera in the construction site coordinate system and each corresponding two-dimensional construction site environment image; wherein Detected feature points with definite identification information and location data are matched as the same and identified as multiple map feature points.

3. The method of claim 1, wherein, The step of determining multiple distinguishable features from the construction site environment as multiple map feature points further includes determining additional map feature points based at least in part on data retrieved from an earthwork information model, wherein the earthwork information model is based on at least one of: a geospatial information system, building information modeling, infrastructure building information modeling, civil information modeling, or a smart city platform.

4. The method of claim 1, wherein, The steps of determining the position and orientation of each received image group and determining the identification information and location data for detected feature points that are matched as the same further include determining a static rating for detected feature points that are matched as the same by utilizing the identification information and location data of at least three detected reference feature points included. in At least one of the following is identified as multiple map feature points: detected feature points that are matched as the same and have defined identification information, location data and static rating, or detected reference feature points that are associated with identification information and location data.

5. The method of claim 2, wherein, Determining identification information and location data for each set of images for the detected feature points that are matched as the same further includes determining a static rating for the detected feature points that are matched as the same by utilizing the position of the at least one camera in the site coordinate system and each corresponding two-dimensional site environment image. in Detected feature points that are matched with the same identifier information, location data, and static rating are identified as multiple map feature points.

6. The method of claim 4, wherein, The method further includes determining an overall rating, which includes at least one of a static rating or a dynamic rating, and determining the dynamic rating individually for each of the plurality of determined map feature points with respect to each positioning camera, wherein the previously determined position and orientation of each of the at least one positioning camera are taken into account.

7. The method of claim 6, wherein, The overall rating is at least one of the static rating or the dynamic rating, and the overall rating is at least two-tiered.

8. The method of claim 7, wherein, If the number of multiple map feature points exceeds a threshold, then at least the map feature point with the lowest overall rating is discarded, wherein the threshold is determined by at least one of manual or automatic methods.

9. The method of claim 1, wherein, The step of determining the position and orientation of earthmoving machinery on the site is further based on tracking navigation feature points between consecutive positioning images, wherein the navigation feature points tracked between consecutive positioning images indicate changes in the position and orientation of the positioning camera.

10. The method of claim 1, wherein, The method further includes determining the location and orientation of the earthmoving machinery in an earthmoving information model, wherein the earthmoving information is based on at least one of the following: geospatial information system, building information modeling, infrastructure building information modeling, civil information model, and smart city platform.

11. The method of claim 10, wherein, The method further includes tools for controlling the earthmoving machinery to create the structure described in the earthmoving information model.

12. The method of claim 10, wherein, The method further includes generating as-built data that will accompany the earthwork information model.

13. The method of claim 11, wherein, The method further includes generating as-built data that will accompany the earthwork information model.

14. The method of claim 12, wherein, The method further includes transmitting the completed data to at least one of a site information management system or machinery operating at the site.

15. The method of claim 13, wherein, The method further includes transmitting the completed data to at least one of a site information management system or machinery operating at the site.

16. The method of claim 1, wherein, The camera's intrinsic parameters are updated by at least one dataset collected during navigation, wherein the dataset is created through a specific segment of the camera path, the specific segment containing feature measurements taken as the camera moves through the path.

Images