Method for determining location and orientation of an earthmoving machine at a work site - Patents.com

Abstract

A method for determining the location and orientation of the earthmoving machine 1 at the work site 14 includes the steps of setting at least one positioning camera PCA above the earthmoving machine, determining a plurality of distinguishable features DF as a plurality of map feature points MFP from the work site environment WE, taking a two-dimensional positioning image PIM by the at least one positioning camera PCA, detecting the distinguishable features DF from the two-dimensional positioning image PIM as navigation feature points NFP, and matching identification information of the navigation feature points NFP with identification information of the plurality of map feature points MFP to determine the location and orientation of the earthmoving machine 1 at the work site 14.

Description

[Technical field]

[0001] The present invention relates to a method for determining the location and orientation of an earthmoving machine on a work site. [Background technology]

[0002] Different types of earth moving machines may be utilized at different ground moving work sites, for example to move earth or rock material from one location to another. Examples of such work sites include, for example, substructure construction work sites, or house construction work sites for buildings, and road construction work sites. Such earth moving machines are, for example, excavators and bulldozers.

[0003] Earth moving machines and their working tools must be able to be positioned and operated very accurately at the work site in order to properly perform the operations for which they are designed. Information regarding the exact location and orientation of the earth moving machine and its tools can be shown to the operator of the earth moving machine so that the operator can use the information when controlling the tools and the machine. This exact location and orientation information of the machine and its tools is particularly important when it is utilized for semi-automatic or fully automatic earth moving machines, i.e. earth moving machines that operate, at least for some time, without constant control by the machine operator, and therefore possible mispositioning of the machine or its tools cannot be immediately corrected by the machine operator.

[0004] Generally, the automatic positioning of the machine can be based on satellite-based positioning systems GNSS (Global Navigation Satellite System), for example GPS (USA), GLONASS (Russia), Galileo (Europe) or Compass (China). Alternatively, the positioning of the earthmoving machine can be provided by a total station that is placed at the work site.

[0005] However, there may not be any or sufficiently accurate satellite-based positioning systems available at all work sites or on all earth moving machines, or the connection to the satellite may be interrupted due to obstacles remaining above or reaching above the earth moving machine. Furthermore, setting up a total station-based positioning system at a work site may be particularly tedious if the total station-based system must be removed from the work site each day or multiple times per day.

[0006] Therefore, an alternative positioning solution is needed. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Bundle Adjustment - A Modern Synthesis, Vision Algorithms: Theory &Practice, B. Triggs, A. Zisser-man & R. Szeliski (Eds.), Springer-Verlag LNCS 1883, 2000 [Non-Patent Document 2] Fischler and Bolles, Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Communications of the ACM, June 1981, volume 24, Number 6 Summary of the Invention [Problem to be solved by the invention]

[0008] SUMMARY OF THE PRESENT EMBODIMENT It is an object of the present invention to provide a novel method for determining the location and orientation of an earth moving machine on a work site. [Means for solving the problem]

[0009] The invention is characterized by the features of the independent claims.

[0010] The present invention is based on the idea of ​​identifying a plurality of distinguishable features from the work site, which are applied as a plurality of unique map feature points, each of which has an identification information including at least a unique identifier such as a feature descriptor, and a location data including at least one of its three-dimensional location in the work site environment or two-dimensional image coordinates with an image identifier. Using the map feature points with the identification information and the location data, a visual localization method for the work machine is realized, where the localization error remains at a controllable, i.e., acceptable level, instead of accumulating over time. Special attention is paid to the georeferencing of the plurality of distinguishable features applied as unique map feature points.

[0011] An advantage of the present invention is that an accurate feature point map is applied to determining the location and orientation of an earth moving machine on a work site.

[0012] Some embodiments of the invention are disclosed in the dependent claims.

[0013] According to an embodiment of a method for determining a location and orientation of an earthmoving machine on a work site, the method includes the steps of: positioning at least one positioning camera over the earthmoving machine, the at least one positioning camera having a determined intrinsic parameter set defining for the image a formation of individual image pixels from a real-world view and a determined location and orientation in the machine coordinate system; determining a plurality of distinguishable features from the work site environment as a plurality of map feature points, the plurality of map feature points having at least identification information and location data, the identification information having at least a unique identifier, and the location data being a three dimensional location at the work site or a two dimensional image coordinate having an image identifier. The method includes at least one of the steps of: taking a two-dimensional positioning image by at least one positioning camera; detecting distinguishable features from the two-dimensional positioning image as navigation features and determining two-dimensional image coordinates and identification information for each navigation feature; and matching the identification information of the navigation features with identification information of at least one of a plurality of map features or detected non-matching features, wherein the step of determining a location and orientation of the earthmoving machine at the work site is based on at least the matched location data of the map features or the detected non-matching features, the two-dimensional image coordinates of the corresponding matched navigation features, and the determined intrinsic parameter sets and the determined location and orientation in the machine coordinate system of each corresponding positioning camera.

[0014] According to an embodiment of a method, a location and orientation of an earthmoving machine in a work site coordinate system is determined, the method comprising: Creating a feature point map, determining a plurality of distinguishable features from the work site environment as a plurality of map feature points by taking a plurality of two-dimensional work site environment images with at least one camera having the determined intrinsic parameter set, the intrinsic parameter set defining a formation of individual image pixels from a real-world view for the images; From each of the multiple 2D work site environment images, Distinguishing features for associating each distinguishable feature with an identity and image coordinates of a respective two-dimensional worksite environment image as a detected feature point; or distinguishable features as reference feature points, the reference feature points being preset at the work site and associated with identification information and location data in the work site coordinate system to further associate each of the reference feature points with image coordinates of a respective two-dimensional work site environment image as a detected reference feature point; detecting at least one of Creating at least one image group, combining the work site environment images together by having at least five of the detected feature points or at least one of the detected reference feature points matched as the same point in the combined work site environment images; receiving at least two worksite environment images and respective image groups including at least three detected reference feature points that are matched as identical points; determining, for each accepted image group, a location and orientation of the image group and identification and location data for the detected feature points matched as the same point by utilizing identification and location data of at least three detected reference feature points included therein, from which the detected feature points matched as the same point and the detected reference feature points matched as the same point are determined as map feature points for a feature map, the map feature points of the feature map having at least identification and location data, the identification information including at least a unique identifier that identifies the distinguishable feature, and the location data including a three dimensional location in a work site coordinate system; and positioning at least one positioning camera above the earthmoving machine, the at least one positioning camera having a determined set of intrinsic parameters and a determined location and orientation in the machine coordinate system; providing a feature points map to an earthmoving machine; capturing two-dimensional positioning images with at least one positioning camera; detecting distinguishable features from the two-dimensional positioning image as navigation feature points and determining two-dimensional image coordinates and identification information for each navigation feature point; matching the identification information of the navigation feature with the identification information of the map feature of the feature map; Including, The step of determining the location and orientation of the earthmoving machine in the work site coordinate system includes at least: location data of matched map features; the 2D image coordinates of the corresponding matched navigation feature points; Determined intrinsic parameter sets and determined locations and orientations in the machine coordinate system of each corresponding positioning camera Based on.

[0015] According to an embodiment of the method, determining a plurality of distinguishable features from the work site environment as a plurality of map feature points includes the steps of taking a plurality of two-dimensional work site environment images by at least one camera having the determined intrinsic parameter set; detecting at least one of the distinguishable features or reference feature points from each of the plurality of two-dimensional work site environment images for associating each of the distinguishable features with an identification and image coordinates of the respective two-dimensional work site environment image as a detected reference feature point, the distinguishable feature or reference feature point being predefined at the work site and associated with an identification and location data for further associating each of the reference feature points with an image coordinates of the respective two-dimensional work site environment image as a detected reference feature point; and creating at least one image group, the distinguishable features or reference feature points being predefined at the work site and associated with an identification and location data for further associating each of the reference feature points with an image coordinates of the respective two-dimensional work site environment image as a detected reference feature point. The method includes the steps of: combining the work site environment images together by having at least one of the detected reference feature points matched as the same point; accepting individual image groups including at least two work site environment images and at least three detected reference feature points matched as the same point; and determining, for each accepted image group, the location and orientation of the image group and identification and location data for the detected feature points matched as the same point by utilizing identification and location data of the at least three detected reference feature points included, wherein at least one of the detected reference feature points matched as the same point with the determined identification and location data, or that match as the same point and associated with identification and location data, is determined as a plurality of map feature points.

[0016] According to an embodiment of the method, determining a plurality of distinguishable features from the work site environment as a plurality of map feature points includes capturing a plurality of two-dimensional work site environment images with at least one camera, the at least one camera having a determined intrinsic parameter set and a determined location in a work site coordinate system; detecting distinguishable features from each of the plurality of two-dimensional work site environment images and determining the distinguishable features as detected feature points using image coordinates of the respective two-dimensional work site environment images; creating at least one image group having matched detected feature points as identical points in the work site environment images; and for each image group determining identification and location data for the matched detected feature points by utilizing the location in the work site coordinate system of the at least one camera having a respective respective two-dimensional work site environment image, wherein the matched detected feature points as identical to the determined identification and location data are determined as the plurality of map feature points.

[0017] According to an embodiment of the method, in the step of determining a plurality of distinguishable features from the work site environment as a plurality of map feature points, the map feature points are determined at least in part from data retrieved from a ground movement information model, the ground movement information model being based on at least one of a Geospatial Information System (GIS), Building Information Modelling (BIM), Infrastructure or substructure Building Information Modelling (I-BIM), Civil Information Model (CIM), or a SmartCity Platform.

[0018] According to an embodiment of the method, the step of determining a plurality of distinguishable features from the work site environment as a plurality of map feature points further includes other map feature points determined at least in part from data retrieved from a ground movement information model, the ground movement information model being based on at least one of a Geospatial Information System (GIS), Building Information Modeling (BIM), Infrastructure or Substructure Building Information Modeling (I-BIM), Civil Information Modeling (CIM) or a SmartCity Platform.

[0019] According to an embodiment of the method, in the step of determining a plurality of distinguishable features from the work site environment as a plurality of map feature points, the map feature points are determined as a combined set of map feature points determined according to the methods disclosed in at least two of claims 1, 2 or 3.

[0020] According to an embodiment of the method, for each accepted image group, the step of determining the location and orientation of the image group and identification information and location data for the detected feature points matched as the same further includes a step of determining a static rating for the detected feature points matched as the same by utilizing the identification information and location data of at least three detected reference feature points included, and at least one of the determined identification information, location data and static rating of the detected feature points matched as the same or the detected reference feature points associated with the identification information and location data is determined as a plurality of map feature points.

[0021] According to an embodiment of the method, the step of determining, for each group of images, identification information and location data for the detected feature points that are matched as the same further includes determining static ratings for the detected feature points that are matched as the same by utilizing locations in a work site coordinate system of at least one camera having a respective respective two-dimensional work site environment image, and the determined identification information, location data and static ratings of the detected feature points that are matched as the same are determined as a plurality of map feature points.

[0022] According to an embodiment of the method, the method further comprises determining an overall rating, the overall rating comprising at least one of a static rating or a dynamic rating, the dynamic rating being determined for each of the plurality of determined map feature points with respect to each individual positioning camera, taking into account its previously determined location and orientation of each of the at least one positioning camera individually.

[0023] According to an embodiment of the method, the overall rating is at least one of a static rating or a dynamic rating, and the overall rating has at least two tiers.

[0024] According to an embodiment of the method, at least the map feature point having the lowest stratum overall rating is discarded if the amount of the plurality of map feature points is greater than a threshold, the threshold being at least one of manually and automatically determined.

[0025] According to an embodiment of the method, the step of determining a location and orientation of the earthmoving machine at the work site is further based on tracking navigational features between successive positioning images, the navigational features tracked between successive positioning images being indicative of changes in location and orientation of the positioning camera.

[0026] According to an embodiment of the method, the method further comprises determining a location and orientation of the earthmoving machine in an earthwork information model, the earthwork information being based on at least one of a Geospatial Information System (GIS), Building Information Modeling (BIM), Infrastructure or Substructure Building Information Modeling (I-BIM), Civil Information Model (CIM) and a SmartCity Platform.

[0027] According to an embodiment of the method, the method further includes controlling a tool of the earthmoving machine to create a structure described in the earthmoving information model.

[0028] According to an embodiment of the method, the method further comprises creating as-built data for the earthworks information model to follow.

[0029] According to an embodiment of the method, the method further includes transmitting the as-built data to at least one of a work site information management system or a machine operating at the work site.

[0030] The present invention will now be described in more detail by way of preferred embodiments with reference to the accompanying drawings, in which: FIG. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a schematic side view of an excavator at a work site. [Figure 2a] 1 is a schematic diagram of an embodiment of a method for determining a location and orientation of an earth moving machine on a work site; [Figure 2b] 1 is a schematic diagram of a system for determining a location and orientation of an earthmoving machine on a work site; [Diagram 3] 1A-1D are schematic diagrams illustrating an example method and system for providing a feature points map for a work site. [Figure 4] FIG. 2 shows a schematic illustration of an example of creating a feature descriptor for a single pixel point in an image. [Figure 5a] FIG. 13 is a schematic diagram illustrating an embodiment for determining map feature points for creating a feature point map. [Figure 5b] FIG. 13 is a schematic diagram illustrating an embodiment for determining map feature points for creating a feature point map. [Figure 5c] FIG. 13 is a schematic diagram illustrating an embodiment for determining map feature points for creating a feature point map. [Figure 5d] FIG. 13 is a schematic diagram illustrating an embodiment for determining map feature points for creating a feature point map. [Figure 6] FIG. 13 is a schematic diagram of another embodiment for determining map feature points for creating a feature point map. [Figure 7a] FIG. 2a and FIG. 2b show a schematic representation of an embodiment for determining the location and orientation of a machine on a work site. [Figure 7b]FIG. 2a and FIG. 2b show a schematic representation of an embodiment for determining the location and orientation of a machine on a work site. [Figure 7c] FIG. 2a and FIG. 2b show a schematic representation of an embodiment for determining the location and orientation of a machine on a work site. [Figure 8] FIG. 1 is a diagram generally disclosing several ground movement information models. [Figure 9] FIG. 2 shows a schematic diagram of an embodiment relating to a visual odometry procedure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] For clarity, the figures illustrate some embodiments of the invention in a simplified manner. Like reference numerals identify like elements in the figures.

[0033] FIG. 1 is a schematic side view of an excavator 1 at a work site 14, where the excavator 1 is intended to be operated. The excavator 1 is an example of an earth moving machine, in connection with which the solution disclosed herein for determining the location and orientation of the earth moving machine at the work site 14 can be applied. The work site 14 comprises at least one three-dimensional area or space, i.e. working environment (WE), in which the active work performed by the earth moving machine will be performed. Depending on the nature or characteristics of the work site 14, in addition to the area or space in which the active work will be performed, the work site 14 can also comprise one or more surrounding areas or spaces that may affect and / or be influenced by the operations to be performed in the area or space in which the active work will be performed. Especially in urban environments, the earth moving machine may also need to temporarily operate outside the actual or officially determined work site. Thus, in this description, the work site 14 may extend over a distance, such as several meters, beyond the area and / or space that is the actual work site 14 or the officially depicted work site 14.

[0034] The excavator 1 comprises a mobile carriage 2 comprising an undercarriage 2a, i.e. a lower carriage 2a and an upper carriage 2b. The lower carriage 2a is provided with caterpillar bands, but could alternatively be provided with wheels. The upper carriage 2b is connected to the lower carriage 2a by means of a rotation axle 3 of the upper carriage 2b. The upper carriage 2b can therefore rotate relative to the lower carriage 2a around a rotation axis 4, as indicated diagrammatically by the arrow R. The rotation axis 4 coincides with the central axis of the rotation axle 3.

[0035] The excavator 1 further comprises a boom 5 connected with the upper carriage 2b such that the boom 5 rotates together with the upper carriage 2b. The boom 5 may comprise at least a first boom section 5a. The boom 5 may also further comprise other boom sections, such as a second boom section 5b. The boom 5 may be raised up and lowered relative to the upper carriage 2b, as indicated diagrammatically by the arrow L.

[0036] The second boom section 5b may be connected to the first boom section 5a by a joint 6, allowing the second boom section 5b to rotate about the first boom section 5a, as shown diagrammatically by arrow T6. At the distal end of the second boom section 5b there is a work tool, in this case a bucket 7, and there may be a joint 8 between the bucket 7 and the second boom section 5b, allowing the bucket 7 to rotate about the second boom section 5b, as shown diagrammatically by arrow T8. Associated with the joint 8 there may also be a joint or mechanism allowing the bucket to be tilted, for example, in a sideways direction.

[0037] Above the carriage 2 there may be a control cabin 9 for an operator 10 of the excavator 1. The control cabin 9 may for example comprise a movable structure by which the vertical position of the control cabin 9 can be adjusted relative to the carriage 2.

[0038] The excavator 1 further comprises at least one control unit 11 arranged to control the operation of the excavator 1, such as the movement of the carriage 2, the boom 5 and the bucket 7 in response to received control actions.

[0039] The excavator 1 may further comprise a number of satellite receiving devices such as one or more antennas 12 if the excavator 1 is intended to be able to make use of some kind of satellite-based positioning system GNSS (Global Navigation Satellite System), i.e. very diagrammatically shown satellites referenced with reference number 13. The one or more antennas 12 may for example be placed on the upper carriage 2b.

[0040] The excavator 1 may further comprise a number of sensors SMUs or a group of sensors SMUs, for example an inertial measurement unit (IMU) for determining the position and / or orientation and / or tilt of the machine and / or the direction in which the machine is facing. The sensor or group of sensors may be at least one of a gyroscope, an accelerometer, an inclinometer, a magnetic compass, an angle sensor, a position sensor, a pendulum, a level measuring device and any other sensor such as a camera sensor, a stereo camera, a laser receiver / detector or a lidar suitable for the purpose of determining at least one of the position, orientation, tilt or the direction in which the object is facing of at least one object attached to each other. In case of an excavator 1 comprising at least one antenna 12 so that a satellite-based positioning system GNSS can be utilized, the at least one sensor may also be a satellite-based compass. The excavator 1 may also comprise a sensor for measuring the distance traveled by the excavator 1, which may be internal or external to the inertial measurement unit IMU.

[0041] FIG. 1 further discloses, in a schematic manner, a number of coordinate systems that may be utilized in the disclosed solution for determining the location and orientation of the excavator 1 on the work site 14. FIG. 1 discloses a machine coordinate system (MCS) that is attached to a point on the excavator 1, which allows the location and orientation of an object to be determined relative to that point in the machine coordinate system MCS. FIG. 1 further discloses a workplace coordinate system (WCS) that is attached to a point on the work site 14, which allows the location and orientation of an object to be determined relative to that point in the workplace coordinate system WCS. FIG. 1 further discloses a world coordinate system (WDCS) that is attached to a point in the real world, which allows the location and orientation of an object to be determined relative to that point in the world coordinate system WDCS.

[0042] FIG. 2a shows a schematic representation of an embodiment of a method for determining the location and orientation of an earthmoving machine at a work site 14. FIG. 2b shows a schematic representation of a system level illustration for determining the location and orientation of an earthmoving machine at a work site 14. The excavator 1 disclosed above is merely one example of an earthmoving machine to which the solutions disclosed herein may be applied. Other earthmoving machines to which the solutions disclosed herein may also be applied are, for example, dozers, wheel loaders, rollers, backhoes, dump trucks, cranes, mobile cranes, forwarders, harvesters, etc. Thus, in the following description, the earthmoving machine may also be referred to using the term "machine 1".

[0043] The method for determining the location and orientation of the machine 1 on the work site 14 comprises the step of positioning at least one positioning camera (PCA), i.e. one or more positioning cameras PCA, above the machine 1. Each one or more positioning cameras PCA has a determined intrinsic parameter set and a determined location and orientation in the machine coordinate system MCS.

[0044] The determined intrinsic parameter set defines the formation of individual image pixels from the real-world view for images captured by each positioning camera PCA. The determined intrinsic parameter set includes parameters defining the camera's behavior. These parameters can include, for example, image center, focal length, skew, and lens distortion. The determined intrinsic parameter set describes to which image pixels in the captured images a specific point in the real-world is set. The determined intrinsic parameter set for each positioning camera PCA can be obtained in advance via laboratory calibration or later by a Structure from Motion procedure, which will be discussed in more detail later.

[0045] The determined location and orientation of the at least one positioning camera PCA in the machine coordinate system MCS must be known in this method. The determined location and orientation can be determined at the time of installation of the respective positioning camera PCA on the machine 1. The determined location and orientation can be a combination of manual and sensor measurements or can be entirely measured by sensors. In any case, it is important that the location and orientation of the respective positioning camera PCA relative to the determined origin and axes (x, y, z) of the machine coordinate system MSC are measured as accurately as possible. The sensor measurements can be determined, for example, either using at least one sensor or measuring device external to the machine 1 and can be available only at the time of installation of the positioning camera PCA on the machine 1 or can be determined using at least one sensor SMU installed on the machine 1 and / or using at least one sensor SMC installed on the positioning camera PCA. At least one sensor SMC, i.e. one or more sensors SMC, in the positioning camera PCA and at least one sensor SMU, i.e. one or more sensors SMC, installed on the machine 1 can be used to determine, for example, the position and / or orientation of the positioning camera PCA and / or the tilt and / or the direction in which the positioning camera PCA is pointing. Regardless of how the location and orientation of the camera in the machine coordinate system MCS is measured, a system for determining the location and orientation of the earthmoving machine at the work site requires the exact location and orientation of each of the at least one positioning camera PCA in the machine coordinate system MCS. The location and orientation of the positioning camera PCA in the machine coordinate system MCS can also be determined or evaluated while operating the machine 1 using at least one sensor SMU in the machine 1 and / or at least one sensor SMC in the positioning camera PCA, so that it can be verified that the location and / or orientation of the at least one positioning camera PCA remains or does not remain as determined for the system, and the location and / or orientation of the at least one positioning camera PCA can be redetermined for the system if it is changed.Furthermore, the machine operator can know if the location or orientation of the positioning camera PCA has changed. The at least one sensor SMC in the positioning camera PCA may for example be at least one of a gyroscope, an accelerometer, an inclinometer, a magnetic compass, an angle sensor, a position sensor, a pendulum, a level measuring device or any other sensor or measuring device suitable for measuring the location and / or orientation or for measuring a change in the location and / or orientation.

[0046] The method for determining the location and orientation of the machine 1 at the work site 14 further includes creating or generating a feature point map, i.e., a navigation map, forming a representation of map features at the work site 14, which will be used to determine the location and orientation of the machine 1 operating at the work site 14. This navigation map is thus a pre-created feature point map of the work site 14 that is subsequently used for real-time navigation of the machine 1 at the work site 14. Several alternative solutions for creating the feature point map will be considered in more detail later in this description.

[0047] The location and orientation of the machine 1 in the work site 14 is determined based at least on the information available in the feature point map and the determined intrinsic parameter set, as well as the determined locations and orientations of the respective corresponding positioning cameras PCA in the machine coordinate system MCS. This step for determining the location and orientation of the machine 1 in the work site 14 is considered in more detail below in connection with Figures 7a, 7b and 7c and the associated description in view of Figures 2a and 2b.

[0048] The necessary data processing for carrying out the determination of the location and orientation of the machine 1 at the work site 14 can be carried out by data processing resources available exclusively in the machine 1, such as the control unit 11, exclusively outside the machine 1, such as data processing resources reserved for the work site information management system, such as a work site server or computer, or available in a cloud service arranged to provide the work site information management system, or by data processing resources available both in the machine 1 and outside the machine 1. Similarly, the feature point map used for determining the location and orientation of the machine 1 at the work site 14 can be located in at least one of the machine 1, the work site information management system or the cloud service. The necessary data transmission can be realized by a dedicated and / or public data transmission network available for the work site 14. In FIG. 2b, the control unit 11, which can be replaced by the work site information management system or by a cloud service, is very diagrammatically represented in FIG. 2b by a box drawn with a dashed line and surrounding the items in the system of FIG. 2b.

[0049] Referring again to the creation or construction of a feature point map already briefly discussed above, Fig. 3 very diagrammatically illustrates a method for providing a feature point map for a work site 14. An embodiment of the method comprises determining a plurality of distinguishable features from the work site environment WE as a plurality of map feature points (MFP), each of the plurality of map feature points MFP having at least identification information and location data. The identification information comprises at least a unique identifier for each map feature point MFP, while the location data comprises at least one of a three-dimensional location of each map feature point MFP in the work site coordinate system WCS, or a two-dimensional image coordinate with an image identifier. The unique identifier may comprise at least some kind of index number for a database and at least one feature descriptor. The feature descriptors identify the map feature points MFP from each other, and thus the feature descriptors are able to identify the individual distinguishable features from each other. Typically, the map feature points MFP have as unique identifiers an index number for the feature descriptor, an ARUCO code, or a checker marker. In other words, to create the feature point map, multiple images are taken of the work site environment WE by at least one camera, these images are then examined or analyzed to find distinguishable features (DF) in these images, and these distinguishable features DF are deemed suitable to provide a map feature points MFP from the work site environment WE to form a feature point map, i.e. a pre-created navigation map that is later used for real-time navigation of the machine 1 at the work site 14.

[0050] Several different solutions for determining the map feature points MFP for forming the feature point map will be considered in more detail later, but before that, some technique aspects related to the discovery and analysis of distinguishable features DF in images taken from the worksite environment WE will be considered.

[0051] To create a feature point map or navigation map, multiple images are taken of the work site environment WE at different positions of the camera / cameras taking the images. The at least one camera in this context may be at least one positioning camera PCA applied to some machines 1 or any other camera CA applied to the work site 14, for example a camera applied to a drone moving in the work site 14. This procedure of taking images from the work site 14 is reminiscent of aerial image taking or aerial image photography, which allows a three-dimensional scene reconstruction of the work site environment WE. However, the perspective of the taken images is preferably substantially the same as the perspective of the at least one positioning camera PCA that will be applied to the machine 1 when the determination of the location and orientation of the machine 1 in the work site 14 is performed. The images are taken with a large amount of overlap in order to combine the images together, i.e. to combine them to form a number of image groups or at least one group of images (GIM). It is noted that each image must overlap sufficiently with at least one other image in the image group GIM, but not with all other images in the image group GIM. The overlap of the images may vary from case to case, for example between 40% and 80%. If the imaged area contains many solid items that remain constant, such as a building with many features, the overlap may be less. On the other hand, if the imaged area contains many moving or variable items, such as crushed stone, young trees, bushes, or grass, or even smooth white walls, the overlap will be much more. The more overlap there is, the more likely the images will be successfully combined, creating a denser feature point map.

[0052] The image data of each of the captured images can be augmented with per-image attitude data and / or GNSS position data if an Inertial Measurement Unit (IMU) and / or GNSS are available, to generate a feature point map during the Structure From Motion workflow, a well-known technique for 3D scene reconstruction, and includes as its main components feature discovery from images, matching of features from and to individual images, motion recovery and triangulation using the camera's intrinsic parameters, and optionally finding optimal model parameters for scene reconstruction via an optimization task called Bundle Adjustment.

[0053] Bundle Adjustment is an optimization task where parameters necessary for 3D scene reconstruction are solved. These may be extrinsic / extrinsic parameters such as the rotation and translation of the images with respect to the other images, the 3D coordinates of common feature points in the images, and the intrinsic parameters of the camera. The extrinsic parameters of the camera may also be the location and pose of the camera in the World WDCS or Worksite WCS coordinate system for example. The mathematical model forming part of the Bundle Adjustment may therefore differ based on the number of parameters applied to the model. It is therefore a design choice that defines what is to be solved. Also, all parameters to be solved or optimized in the model may be assigned preliminary initial values ​​that are specified using the initial values, while some of the parameters in the model may have fixed values ​​that are known in advance. The purpose of the Bundle Adjustment is to optimize all the model parameters selected to be optimized, such as the extrinsic image parameters that are related to the information of features present in the captured images. If the intrinsic and extrinsic camera parameters considered above are already known, Bundle Adjustment is not necessary. It may be the case that it is not always meaningful to include all matched detected feature points (DFP) in the Bundle Adjustment optimization. The discarded detected feature points DFP can be reproduced in the final triangulation after Bundle Adjustment. These detected feature points DFP can then be utilized when navigating using a PnP based method. Also, since image extrinsic parameters are resolved within the Bundle Adjustment, even detected feature points not matched (DFP-NM) that are present on the image can be utilized in determining the machine location and orientation in real time, in an embodiment as will be considered later, and therefore can be stored with the MFP without 3D coordinates.It should be noted that when speaking of position and / or orientation / pose, intrinsic parameters of the image, extrinsic parameters of the image or parameters of the image, these are similar parameters of the camera at the time of taking the image. Thus, for example, if the intrinsic and extrinsic parameters of the camera are known at the time the image is taken, these parameters can be used as the intrinsic and extrinsic parameters of the image taken by the camera, and if the intrinsic and extrinsic parameters of the camera are not known at the time the image is taken, they will be solved in the Bundle Adjustment and will also be considered as the intrinsic and extrinsic parameters of the image. Alternatively, if only the intrinsic camera parameters are known, they can also be provided to the Bundle Adjustment as known information. As indicated above, the Bundle Adjustment procedure is a widely known technique in connection with 3D scene reconstruction, and for example Non-Patent Document 1 discloses the Bundle Adjustment procedure in more detail.

[0054] Usually, there is no image attitude and / or GNSS position data available when images are taken from the work site environment. For example, GNSS is not available in tunnels, under bridges, or in urban environments with tall buildings, or the camera CA, PCA taking the images is not connected to a positioning system that would determine the location and / or orientation of that camera CA, PCA. In that case, a ground control point network can be applied. Ground control points are reference points or reference features that are preferably uniformly spread over the entire work site 14. As these reference points need to be recognized from the images, they are equipped with specific markers to enable their recognition. Ground control points can be, for example, 4-square checker markers or Aruco markers, the former being usually more accurate to detect from the images. If checker markers are used, the markers do not necessarily have to be unique, i.e. they can all be exactly the same, so that the location of the individual markers identifies each marker. Although the checker markers can be detected in the image, their actual identification is performed by context-based recognition, e.g. using computer readable numbers on the marker, i.e. those surrounding the object visible in the image, or later detected in the 3D scene using combined mathematics of the distance and angle of the marker between points in both the workplace coordinate system and the 3D coordinate system in the reconstruction of the image group.

[0055] The GNSS position data with possible attitude data for each image, or alternatively the ground control point network, is applied to create a feature point map, i.e. a navigation map, to bring the work site environment scene into the work site coordinate system WCS at the correct scale. Alternatively, the attitude data for each image and the GNSS position data can be applied together with the ground control point network to create a feature map, which brings the work site environment scene into the work site coordinate system WCS at the correct scale. When applying the bundle adjustment, the coordinate transformation from the reconstructed 3D work site environment scene to the world WDCS or work site WCS coordinate system can be solved in a separate optimization task after the bundle adjustment. A covariance matrix describing the accuracy information for the detected features can also be solved in the bundle adjustment procedure.

[0056] In order to reconstruct a 3D scene from captured images of the workplace environment, it is necessary to find common feature points in multiple images. The 3D coordinates for these common feature points can then be calculated. The more common feature points found in the captured images, the more likely it is that a high-precision reconstruction will be achieved.

[0057] In the captured images, the features in the 3D workplace environment WE are depicted on a 2D plane and the 2D location of the features in the images is determined by the 2D image coordinates, i.e. image pixel coordinates. The discovery of the same feature from several images can be realized fully automatically and there is no need to manually pick common points of interest in the images, which is not excluded here. SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Feature) and ORB (Oriented FAST and Rotated BRIEF) are some examples of these feature detection algorithms. All of these create a respective feature descriptor for a single pixel point in the image by utilizing the pixel values ​​or pixel textures around that single pixel point. The feature descriptor determines or defines the characteristics of the detected feature. A prime example of creating a feature descriptor for a single pixel point in an image by utilizing the pixel values ​​around that pixel point is shown in FIG. 4, which shows that the feature descriptor for pixel point p will be created by the pixel values ​​or textures of the remaining 27 pixels around pixel point p.

[0058] The use of ORB-based feature descriptors is advantageous for determining the location and orientation of a mobile machine 1 because the matching ORB-based feature descriptors are substantially robust to changes in imaging conditions such as lighting conditions, blur, rotation and scale, and also compute fast enough for real-time applications.

[0059] Identical features appearing in different images can be matched by comparing their feature descriptors. According to an embodiment, the feature matching procedure can be realized using the well-known brute force search based problem solving technique and an algorithmic paradigm that consists in systematically enumerating all possible candidates for the solution and checking whether each candidate satisfies the problem statement.

[0060] Thus, in the solution disclosed herein, the procedure related to matching features appearing in different images can include comparing the descriptors of all features and selecting the best match to correspond to the same feature in the different images. Furthermore, a forward-backward consistency check can be performed, for example verifying whether the i-th feature descriptor in the first image has the j-th feature descriptor in the second image as a best match, and vice versa whether the j-th feature descriptor in the second image has the i-th feature descriptor in the first image as a best match. Finally, only feature descriptors that match a sufficiently short feature descriptor distance, which is the abstraction distance defined for the feature descriptors, are accepted to represent the same feature in different images.

[0061] Once the 3D location coordinates for the features in the world WDCS or work site WCS coordinate system and their respective 2D image coordinates in the images where the features are visible, along with their poses and positions for all images and feature descriptors, the feature descriptors and their 3D coordinates can be linked together to create a feature point map or navigation map. In the feature point map, each individual feature is preferably uniquely indexed. In the feature point map for the solution disclosed herein, the following information can be stored, for example, for each feature forming the map feature point MFP: Feature index (e.g. index number starting from 0 or 1) Feature Descriptor (such as ORB Descriptor or ARUCO Code) 3D worksite WCS or World WDCS coordinate system coordinates, possibly including an ellipsoid indicating the uncertainty of the accuracy of the determined coordinates An image identifier that identifies those images in which the same feature is visible (there is at least one, and typically at least two, images in which the same feature is visible) Image coordinates for the location of the feature relative to those images where the feature can be seen, possibly including an ellipse indicating the uncertainty in the accuracy of the image coordinates Position and orientation of each image in the World WDCS or Worksite WCS coordinate system Covariance matrix, i.e. accuracy information for features ·Static rating characteristics Semantic classification information including man-made or natural object classification describing the location of features, i.e. balcony, construction shed, manhole, thick tree, broken stone All directions, north in degrees, north-east-west directions, or observation directions that allow you to see features such as the ground area of ​​the work site, i.e. the south parking area

[0062] First embodiment for providing map feature points for creating a feature point map 5a, 5b and 5c are schematic diagrams of an embodiment for determining a plurality of map feature points MFP for creating a feature point map, i.e., a first embodiment for determining a plurality of distinguishable features DF as a plurality of map feature points MFP from a work site environment WE, the plurality of map feature points MFP having at least identification information and location data, the identification information including at least one unique identifier such as a feature descriptor for the map feature point MFP for identifying the map feature point MFP from an image, and the location data including at least one of the following information: a three-dimensional location of the map feature point MFP in the work site 14, i.e., in the work site coordinate system WCS, or a two-dimensional image coordinate with an image identifier, i.e., at least one image in which the map feature point MFP is found and from which an image pixel location is found. In this embodiment, the map feature points MFP for the feature point map are determined by utilizing a reference feature point (RFP) that is preset in the work site 14.

[0063] The embodiment of Fig. 5a, Fig. 5b and Fig. 5c includes taking a plurality of two-dimensional worksite environment images (WIM) by at least one camera CA, PCA, the at least one camera CA, PCA having a determined intrinsic parameter set. Fig. 5b shows a schematic of four successive worksite environment images WIM taken from the worksite environment WE. The worksite environment images WIM are preferably taken such that there is a significant overlap between successive images WIM, for example 40% to 80% overlap. For clarity, the successive images WIM are shown separated from each other in Fig. 5b. The leftmost image WIM can be considered as the first taken worksite environment image WIM and the rightmost image WIM can be considered as the last taken worksite environment image WIM.

[0064] The taking of the images WIM can be performed by at least one positioning camera PCA mounted in any machine 1 moving around the work site 14 and / or by at least some other camera CA which is not a positioning camera PCA mounted on any machine 1. The at least one other camera CA can for example be mounted on a drone adapted to move in the work site environment WE. The one other camera CA is shown very diagrammatically in Fig. 1. The focus of the camera CA, PCA is shown diagrammatically in Fig. 5b by a black dot, indicated with the reference CFP.

[0065] The step of capturing a plurality of two-dimensional work site environment images WIM includes capturing from each of the plurality of two-dimensional work site environment images WIM: a) distinguishable features DF for associating each distinguishable feature DF with the identity and image coordinates of the respective two-dimensional work site environment image WIM as detected feature points DFP, or b) reference feature points RFP, which are preset in the work site 14 and are associated with identification information and location data in order to further associate each reference feature point RFP with the image coordinates of the respective two-dimensional work site environment image WIM as a detected reference feature point (DRFP); This is followed by detecting at least one of:

[0066] Thus, the step of detecting distinguishable features DF from individual images of the plurality of images may include only one of options a) and b), or both options a) and b), for example only finding features of option a) from some images, only finding features of option b) from some images, and also finding both features of option a) and option b) from some images.

[0067] The acquired two-dimensional work site environment image WIM can therefore include two types of distinguishable features, namely, a first feature determined as a detected feature point DFP, and a second feature being a reference feature point RFP determined as a detected reference feature point DRFP.

[0068] Considering first the reference feature points RFP, i.e. option b) above, the reference feature points RFP are reference points that are pre-set on the work site 14. They may be, for example, markers as discussed above. In Fig. 5b, these reference feature points RFP correspond to a graphic symbol that includes a circle encircling a cross. The lines of sight from the camera focal point (CFP) to the reference feature points RFP are indicated diagrammatically using respective arrows. In the work site environment image WIM, the reference feature points RFP are set at the locations indicated by the asterisk graphic symbol.

[0069] Once the reference feature points RFP are associated with their identification and location data in the three-dimensional world WDCS or work site WCS coordinate system coordinates and associated with their respective image coordinates in the respective two-dimensional work site environment image WIM, the reference feature points RFP can be determined as the detected reference feature points DRFP. Thus, the asterisk graphic symbols in Figure 5b can be represented as the detected reference feature points DRFP, and the image coordinates represent the image pixel coordinates of the asterisk associated with the respective detected reference feature point DRFP in the respective work site environment image WIM.

[0070] Considering next the distinguishable features DF determined as the detected feature points DFP, FIG. 5b discloses a building 15 associated with three triangles with tips pointing up, and two stones 16 associated with a triangle with tips pointing up, as well as one stone 16 associated with a triangle with tips pointing down. The triangles in FIG. 5b depict distinguishable features DF detected in the work site environment image WIM and are different from the pre-defined reference feature points RFP discussed above. The downward pointing triangles describe distinguishable features DF that cannot be matched as the same or matched as the same in two or more images, as will be discussed in more detail below. In FIG. 5b, the triangles are shown to be associated with objects that are considered immobile in the work site 14, and these types of objects are preferred from the perspective of creating a feature point map for maneuvering the machine 1. Consideration of immobile or mobile objects in the work site 14 can be performed using semantic image segmentation as known in the art. For any point, consideration of stationary or mobile objects in the work site 14 can also be performed manually, for example by selecting all distinguishable features detected from a vehicle soon to leave the work site 14 and determining them to be present on a mobile object. The lines of sight from the camera focus CFP to the detected triangles are shown diagrammatically using respective arrows. In the work site environment image WIM, these triangles are set at the locations indicated by the small black circle symbols.

[0071] When the distinct features DF represented by the triangles are associated with their identities and their image coordinates in each two-dimensional work site environment image WIM, they are determined as detected feature points DFP. Thus, the small black circle symbols in Fig. 5b can be represented as detected feature points DFP, and the image coordinates represent the image pixel coordinates of the small black circle associated with each detected feature point DFP in each work site environment image WIM.

[0072] It is noted herein that the procedure of detecting the distinguishable features DF and the determination of the detected feature points DFP and the detected reference feature points DRFP are performed as long as a new work site environment image WIM is taken, which does not necessarily have to be in real time. It is also noted herein that some reference feature points RFP may initially be erroneously detected as detected feature points DFP, but may later be correctly interpreted as detected reference feature points DRFP.

[0073] When a work site environment image WIM is acquired and a distinguishable feature DF is detected therein, the distinguishable feature DF is matched with the same detected feature point DFP or the same detected reference feature point DRFP in a different work site environment image WIM. - detected feature points DFP, or - Detected reference feature points DRFP At least one group of images GIM is also created from the work site environment image WIM by combining the work site environment image WIM with at least one of the following:

[0074] When creating at least one image group GIM from the work site environment image WIM, at least one image group GIM, i.e. at least one image collage, is created from the work site environment image WIM based on certain conditions considered below, in which at least two different work site environment images WIM taken from different camera locations are grouped into the same group, i.e. image group GIM based on detected feature points DFP and / or detected reference feature points DFRP that match as the same points in these different work site environment images WIM. At least two work site environment images WIM can be combined into an acceptable image group GIM based on the following condition: each of these work site environment images WIM selected into the image group GIM has at least five detected feature points DFP or detected reference feature points DRFP that match as the same point between each single work site environment image WIM and one other single image in the remaining image group, and the entire acceptable image group GIM contains at least three detected reference feature points DRFP that match as the same point in all work site environment images WIM of the acceptable image group GIM. More specifically, when combining the next image into the image group, there must be at least five points (either detected feature points DFP or detected reference feature points DRFP, or some kind of mixture of them such as four DFPs and one DRFP) that combine the next image into the image group, and these points must be present in one single image already present in the image group. Therefore, these five points do not have to be present in all other images of the image group. One image with five connection points is sufficient, two or more are acceptable as well, but as already mentioned, this is not necessary.

[0075] For example, an image group can include 100 images, which are taken from all around a building. Each of these images can include five points that connect the image to the previous image (except the first image) and the following image (except the last image, which can be connected to the first image). Another example can be an example where there are four images in the image group GIM, where the second image is connected to the first image by five points, the third image is connected to the first image by five points, and the fourth image is connected to the third image by five points and five points are connected to the second image. Thus, the fourth image is connected by five points (either the detected feature points DFP or the detected reference feature points DRFP, or a combination thereof) to the two images in the image group that are already connected prior to the connection of the fourth image.

[0076] When creating the image groups mentioned above, no information external to the work site environment images WIM is needed to make the decision to combine the respective work site environment images WIM to form the image group GIM, however, if there is information available about where each of the images are taken, such as the location and orientation of the camera when these images are taken, this information can be used to speed up the combining of these images, since it can tell that some images do not even have a theoretical possibility of having a single coincidence, for example if the images are taken in opposite directions from the same location.

[0077] Referring to Figure 5b, this means that, at a minimum, an acceptable image group can be formed using the three leftmost images, or the rightmost image with the second-from-the-left image. The best image group would include all four of the workplace example images of Figure 5b.

[0078] FIG. 5c shows a schematic representation of an image group GIM formed of five different work site environment images WIM, which may or may not be consecutive images WIM taken by at least one camera CA, PCA. However, knowing the order of the images can speed up the grouping of images. In FIG. 5c, which differs slightly from FIG. 5b for clarity, the triangles with tips pointing up are now simply represented as detected feature points DFP that are matched as the same in at least two work site environment images WIM, and the circles surrounding the crosses are now simply represented as detected reference feature points DRFP that are matched as the same in at least two work site environment images WIM. FIG. 5c further includes triangles with tips pointing down, which represent distinguishable features DF detected in the work site environment images WIM, but which have not been identified, i.e. have not been matched as the same point in more than one work site environment image WIM.

[0079] In the example of FIG. 5c, each single work site environment image WIM happens to contain at least one detected reference feature point DRFP. This situation can also be the situation where one, two or even three of the images do not contain a single detected reference feature point DRFP. The five different work site environment images WIM in the example of FIG. 5c are joined together, i.e. overlapping and connected together in at least one direction in the plane of the paper, to form a group GIM of work site environment images. This group is acceptable because there are at least three reference feature points RFP that match as the same point in at least two images, and each image is joined to some other image that has at least five detected reference feature points DRFP or at least one of the detected feature points DFP that match as the same point. It should be noted that the loss of any one of these images may prevent the group from forming an acceptable image group, since all of these images are required to join the images together and to have at least three reference feature points RFP that match as the same point in at least two images.

[0080] Such an individual group, i.e. an individual image group GIM that includes at least three detected reference feature points DRFP that match as the same and that connects the individual image to at least one other image of the group by at least five points that match as the same, is accepted for further analysis to determine map feature points for the feature point map. If the condition is not met, the image group GIM is rejected from further analysis. There is still an opportunity to avoid the rejection by determining the three-dimensional location of a sufficient amount of detected feature points DFP that match as the same, or any other such points from the images, which can be detected from at least two images, and whose three-dimensional coordinates can be determined to be the detected reference feature points DRFP to meet the minimum amount of at least three detected reference feature points DRFP that match as the same. This kind of operation can save time, since it is not necessary to take another set of images with visible markers. The number of images can be much more than five in some cases.

[0081] Referring again to the example of Fig. 5b already mentioned above, a second work site environment image WIM from the left and rightmost work site environment images WIM can form two acceptable image groups, since both images refer to the same three detected reference feature points DRFP and therefore also contain in the image group at least three detected reference feature points DRFP that are matched as the same, and also both images refer to at least two detected feature points DFP that are matched as the same, thus fulfilling a total of five DFP / DRFP points that are matched as the same. In this case, there are three different detected reference feature points DRFP in common to these two images. Similarly, for example, the three leftmost images or the three rightmost images can together form two respective acceptable image groups. However, for example, the two rightmost images cannot form an acceptable image group without some other image, even though they have five additional detected feature points DFP matched as the same, because the two rightmost images only have the two same detected reference feature points DRFP that they reference and therefore do not satisfy the condition of having at least three detected reference feature points DRFP matched as the same in an image group.

[0082] In a further analysis step, by utilizing the identification information and location data of at least three detected reference feature points DRFP that match as the same point contained in the accepted image group GIM, the location and orientation of the image group GIM and the identification information and location data for the detected feature points DFP that match as the same point are determined for each accepted image group GIM, - the detected feature points DFP that match the determined identity and location data as the same points, or - Detected reference feature points DFRP that are matched as the same point and associated with identity and location data At least one of the three-dimensional locations of the detected reference feature points DFP of each image group GIM is determined as a plurality of map feature points MFP. In this procedure, the three-dimensional locations of the detected feature points DFP of each image group GIM are determined in the work site coordinate system WCS by using the three-dimensional locations of each of the at least three detected reference feature points that are matched as the same point.

[0083] In other words, only these detected feature points DFP and detected reference feature points DRFP in the accepted image group GIM can be selected as map feature points MFP for the navigation map, and these detected feature points DFP and detected reference feature points DRFP are matched as the same point based on their determined identification information determined in the step of detecting distinguishable features DF from each of the multiple two-dimensional work site environment images WIM. Figure 5d is a schematic illustration of an example of accepted map feature points MFP correspondingly selected from the detected feature points DFP and detected reference feature points DRFP shown in Figure 5c.

[0084] According to an embodiment related to the determination of map feature points MFP disclosed in the present specification, the step of determining, for each accepted image group GIM, the location and orientation of the image group GIM and the identification information and location data for the detected feature points DFP that are matched as the same further includes a step of determining a static rating for the detected feature points DFP that are matched as the same by utilizing the identification information and location data of at least three detected reference feature points DFRP included in the accepted image group; - the detected feature points DFP that match the determined identity, location data and static ratings, or - Detected reference feature points DRFP associated with identity and location data At least one of the plurality of map feature points MFP is determined.

[0085] The static rating provides a measure of the accuracy of the location of the detected feature points DFP applied as the map feature points MFP. This accuracy information is determined by utilizing the identification information and location data of at least three detected reference feature points DRFP. With respect to the accuracy of each detected feature point DFP, for example, the distance of the detected feature point DFP from the camera at the time the image was taken can have a large effect on the way in which the retrieved information becomes less accurate, for example, the further the detected feature point DFP is from the camera. Most of the information related to the static rating of the detected feature points DFP is determined during the bundle adjustment discussed above and is represented by the corresponding covariance matrix for the three-dimensional location of each detected feature point DFP. The static rating may therefore be an ellipsoid that indicates the uncertainty of the accuracy of the determined three-dimensional coordinates.

[0086] Because the detected reference feature points DFRP correspond to markers previously set in the work site 14, their location data is known in advance, and therefore the static rating for the detected reference feature points DFRP is considered to be 100% or very close to 100%. At its best, the static rating for the detected feature points DFP can approach 100% with very accurate calculations during the bundle adjustment, but typically the accuracy for the detected feature points DFP is somewhat less than 100%.

[0087] Consideration of stationary or moving objects in the work site 14 can be performed using semantic image segmentation and will affect the static ratings determined for the detected feature points DFP on the objects. For some points, semantic image segmentation can also be performed manually, for example by selecting all distinguishable features or detected feature points DFP detected from a car that will soon leave the work site 14 and determining that they are on a moving object, which will ultimately result in the static ratings for those points being considered the lowest and therefore not being selected as map feature points MFP. However, these points can be used when combining the images into an image group GIM. In this case, it must be kept in mind that once the car has left, this area will lack map feature points MFP. Considerations related to the ratings can be found later in this description.

[0088] Second embodiment for providing map feature points for creating a feature point map 6 is a schematic diagram of a second embodiment for determining a plurality of map feature points MFP for creating a feature point map, i.e., determining a plurality of distinguishable features DF as a plurality of map feature points MFP from a work site environment WE, the plurality of map feature points MFP having at least identification information and location data, the identification information including at least one unique identifier such as a feature descriptor for the map feature point MFP for identifying the map feature point MFP from an image, and the location data including at least one of a three-dimensional location of the map feature point MFP in the work site 14 or a two-dimensional image coordinate with an image identifier. In this embodiment, the three-dimensional location of the map feature point MFP for the feature point map is determined by utilizing a satellite-based positioning system GNSS with a camera CA, PCA applied with at least one possible sensor for determining the position of the camera. Once the orientation information of the camera CA, PCA is known, it can be used together with the position / location information.

[0089] This embodiment includes the step of taking a number of two-dimensional work site environment images WIM by at least one camera CA, PCA, with at least one camera CA, PCA having a determined intrinsic parameter set and also with a determined location in the work site coordinate system WCS, and possibly a determined orientation. As already discussed, the work site environment images WIM are preferably taken such that there is a significant overlap between successive images WIM, for example an overlap of 40% to 80%. The taken work site environment images WIM may be similar to the work site environment image WIM shown in FIG. 5b, but with different possible pre-set reference feature points RFP at the work site 14, and what can be seen in the taken work site environment images WIM can be considered as distinguishable features similar to the points associated with the detected feature points DFP in FIG. 5b. In other words, in this embodiment, the reference feature points RFP are not required as reference feature points with a pre-determined three-dimensional location, but can be used if desired.

[0090] The acquisition of the images WIM can be performed by at least one positioning camera PCA mounted in any machine 1 moving around the work site 14 and / or by at least some other camera CA that is not a positioning camera PCA mounted on any machine 1. Also in this specification, the at least one other camera CA can be mounted on, for example, a drone adapted to move in the work site environment WE, as long as at least the location of the camera CA can be determined with respect to the respective image WIM. If determined, the orientation information of the camera CA can also be used.

[0091] The step of capturing a plurality of two-dimensional work site environment images WIM is followed by a step of detecting distinguishable features DF from each of the plurality of two-dimensional work site environment images WIM and determining the distinguishable features DF as detected feature points DFP using image coordinates of each of the two-dimensional work site environment images WIM.

[0092] Thus, in the present specification, the two-dimensional work site environment image WIM obtained includes only one type of distinguishable feature, namely the features considered to correspond to the detected feature points DFP having a triangle with a tip pointing up or down as described in Fig. 5b and Fig. 5c. Also, as mentioned above, possible pre-defined reference feature points RFP in the work site 14, which are visible in the work site environment image WIM, are also considered as detected feature points DFP when detected, and again no location information from them is necessary, but can be used if necessary. Thus, in the present specification, the disclosure of Fig. 5b and Fig. 5c can be applied if the graphic symbols associated with the reference feature points RFP and the detected reference feature points DRFP in the figures are either omitted or considered to be replaced by a triangular graphic symbol associated with the detected feature points DFP. When the distinguishable features DF represented using triangles are associated with their image coordinates, i.e. their image pixel coordinates, in the respective two-dimensional work site environment image WIM, they are determined as detected feature points DFP.

[0093] It is noted that in this specification, the procedure of detecting distinguishable features DF and detected feature points DFP is carried out as long as a new work site environment image WIM is acquired, which does not necessarily have to be in real time.

[0094] Upon obtaining the work site environment image WIM and detecting the distinguishable features DF therein, at least one image group GIM is also created from the work site environment image WIM by combining the work site environment images WIM having the same matched detected feature points DFP in the different work site environment images WIM. The procedure herein for creating at least one image group is similar to the procedure disclosed above, except that here only the detected feature points DFP are considered.

[0095] Further, to determine map feature points MFP for the feature point map of the work site 14, for each image group GIM, identification information and location data for the detected feature points DFP that match as the same are determined by utilizing the location and orientation in the work site coordinate system WCS of at least one camera CA, PCA having an individual respective two-dimensional work site environment image WIM, and the detected feature points DFP that match as the same as the determined identification information and location data are determined as multiple map feature points MFP.

[0096] In other words, only these detected feature points DFP in the image group GIM will be selected as map feature points MFP for the feature point map, and these detected feature points DFP will be matched as the same points based on their determined identification information determined in the step of detecting distinguishable features DF from each of the multiple two-dimensional worksite environment images WIM.

[0097] According to an embodiment related to determining map feature points MFP disclosed in this specification, the step of determining, for each image group GIM, identification information and location data for the detected feature points DFP that are matched as the same further includes a step of determining static ratings for the detected feature points DFP that are matched as the same by utilizing the location and orientation in the work site coordinate system WCS of at least one camera CA, PCA having each respective two-dimensional work site environment image WIM, and the detected feature points DFP that are matched as the same as the determined identification information, location data and static ratings are determined as multiple map feature points MFP.

[0098] Here, the considerations related to the static rating can also be applied, except that here, instead of the identification information and location data of at least three detected reference feature points DRFP, the static rating is determined by utilizing the location and possibly also the orientation in the work site coordinate system WCS of at least one camera CA, PCA having an individual respective two-dimensional work site environment image WIM. In other words, when an image is taken, the location and orientation data of the camera CA, PCA that took the image and their accuracy information, i.e., for example, the determined accuracy prediction value, can be used at the exact time instant when the image is taken. The static rating can further include detected feature points DFP accuracy information derived from the image location / orientation measurement and the bundle adjustment calculation.

[0099] Differences in the static ratings of map feature MFPs are due, for example, to where the map feature MFP is located and the distance between the cameras CA and PCA at the time the image was taken, as well as the type of distinguishable feature DF that the map feature MFP becomes. If the map feature MFP is located on a thin tree or thin branch, the semantic classification can give it a lower rating compared to a thick tree or thick branch detected in approximately the same situation. In the exact same situation, a map feature MFP placed in the corner of a window of a building can be given an even higher rating. Furthermore, when conditions change, the superiority of the static ratings of map feature points MFP located on building window corners, thin trees, thin branches, thick trees or thick branches can vary greatly, for example, due to differences in the distance and / or accuracy of these objects from the cameras CA, PCA at the time these images were taken, and differences in weather or lighting conditions at the time each image was taken, or differences regarding each object, i.e., whether each of these objects was imaged in cloudy conditions, sunny conditions, foggy conditions, or rainy conditions. In short, there are many variables that affect the static rating. Information related to the static rating and the static rating itself can be included in the identification information of each map feature point MFP.

[0100] The rating is useful in situations where there are many map feature points MFPs available, since the determination of the location and orientation of the machine can be performed using the map feature points MFPs with higher ratings, i.e. by at least temporarily discarding the map feature points MFPs with lower ratings at the current location. A rating value threshold can be selected, for example, if map feature points MFPs with too small values ​​are not used for determining the location and orientation of the machine 1. Another approach may be by selecting the amount of the map feature points MFPs with the highest ratings that are located in the area where the positioning camera PCA is guided relative to the most recently determined location and orientation of the positioning camera PCA. The approach that takes into account the already determined location and orientation of the positioning camera PCA can also use dynamic rating, which will be considered later.

[0101] According to an embodiment, a total rating for the detected feature points DFP is further determined, the total rating including at least one of a static rating or a dynamic rating, i.e. a static rating and / or a dynamic rating, the dynamic rating being determined for each of the plurality of determined map feature points MFP, taking into account the previously determined location and orientation or facing direction of each of the at least one positioning camera individually with respect to the respective positioning camera PCA. The facing direction or orientation of each of the at least one positioning camera PCA here means the viewing angle sector of the respective positioning camera PCA. If a large number of map feature points MFP are still not close to the observation area observed by the positioning camera PCA, these map feature points MFP will be able to obtain the lowest dynamic rating for that positioning camera PCA, which may be for example 0, i.e. zero or empty. The total rating may be the sum of the static rating and the dynamic rating, or the total rating may be the product of the static rating and the dynamic rating. Thus, for example, a map feature point located behind one positioning camera PCA may get an overall rating of zero, which may mean that when determining the location and orientation of the machine, no attempt is made to find such map feature point MPF from the images taken at that time by the respective one positioning camera PCA. On the other hand, if there is another positioning camera PCA located on the other side of the machine 1, whose orientation or facing direction is different by about 180 degrees to the above-mentioned "one positioning camera PCA", this other positioning camera PCA will probably give a dynamic rating of 1 for each map feature point MPF, i.e., a total rating or the highest rating. Furthermore, there may be a chance that the map feature point MPF in question is between two positioning cameras PCA, and therefore the dynamic ratings of both cameras may be 0, i.e., zero or empty.

[0102] According to this embodiment, as discussed above, a dynamic rating is determined for each of the plurality of determined map feature points MFP by individually considering the already determined location and orientation of each of the at least one positioning camera PCA for each individual positioning camera PCA. Additionally or alternatively, herein, a different weight or dynamic rating can be applied for each map feature point MFP based on the realized distance of each positioning camera PCA from the detected feature point DFP corresponding to that map feature point MFP.

[0103] The overall rating is then either the static or dynamic rating or a combination, or a weighted combination of the static and dynamic ratings, determined for the map feature point MFP and each of the positioning cameras PCA.

[0104] According to an embodiment, the overall rating is at least one of a static rating or a dynamic rating, and the overall rating has at least two tiers.

[0105] According to this embodiment, the overall rating is a static rating or a dynamic rating, or a combination of a static rating and a dynamic rating. Furthermore, the overall rating is at least two tiers, i.e. includes at least two different levels, and can be classified into different rating classes, e.g., accuracy or other data related to the individual map feature points MFP associated with each detected feature point DFP and their point availability at each time instant, or can be classified into a rating based on, e.g., accuracy and other identification information of the detected feature points DFP corresponding to the map feature points MFP, and the location of the individual map feature points MFP relative to the individual positioning camera PCA and orientation, i.e., its field of view sector. The map feature points corresponding to the detected reference feature points DRFP, due to the pre-set markers at known positions in the work site 14, usually have the highest accuracy and therefore are usually classified as having the highest tier rating.

[0106] According to an embodiment, at least the map feature point MFP having the lowest tiered overall rating is discarded if the amount of the map feature points is greater than a threshold, the threshold being manually determined and / or automatically determined.

[0107] According to this embodiment, if the number of applicable map feature points MFP applied to the feature point map is unnecessarily high, map feature points MFP with low accuracy can be discarded as also previously described in relation to the nominal usefulness. This reduces the computational power required to determine the location and orientation of the machine 1 and / or improves the rapidity of determining the location and orientation of the machine 1. The determination of the threshold value can be based on, for example, historical data regarding the number of reference points DFP detected in the work site 14 and other relevant data, and the historical data can be updated based on data received during the currently ongoing work phase.

[0108] Third embodiment for providing map feature points for creating a feature point map Disclosed herein is a third embodiment for determining a plurality of map feature points MFP for creating a feature point map, i.e., a third embodiment for determining a plurality of distinguishable features DF as a plurality of map feature points MFP from a work site environment WE, the plurality of map feature points MFP having at least identification information and location data, the identification information including at least one unique identifier such as a feature descriptor for the map feature point MFP, and the location data including at least a three-dimensional location of the map feature point MFP in the work site 14.

[0109] According to this third embodiment, the map feature points MFP for creating the feature points map is at least partially determined from data retrieved from a ground movement information model, the ground movement information model being based on at least one of a geospatial information system (GIS), building information modeling (BIM), infrastructure or substructure building information modeling (I-BIM), civil information modeling (CIM) or a SmartCity Platform. The ground movement information model includes a classification system for the work site 14 that can define the meaning and characteristics of different types of substructures and their models and different substructures, such as different layers of the foundation structure, infill layers, surface layers, etc. In the ground movement information model, the planned final structure may be divided into many substructures that are to be completed to complete the planned final structure. In turn, each work phase may include specific work orders, even in the form of many specific individual operations to be performed by the machine 1 to complete the work phase.

[0110] According to this embodiment, the ground movement information model may include at least some distinguishable features DF whose three-dimensional location is known in the model, for example based on some geometric information of the structures in the model, and which can be retrieved from the model to be used as the basis for a new map feature point MFP. The basis for the new map feature point MFP may include information in the identification information and location data, but the identification information may not include the feature descriptors necessary to make the feature a map feature point MFP. The feature descriptors may be generated by taking at least one image including the feature area and its surroundings. Another method may be to identify features that can identify the location in the ground movement information model, i.e., earthwork information model, from the generated map feature point MFP or from the distinguishable features DF determined from the captured multiple two-dimensional work site environment images WIM, and generate feature descriptors from the location data for these features from the earthwork information model, and from at least one image in which these features can be found as distinguishable features. If the feature descriptor for a feature is obtained from distinct features found in only one image, then the generated map feature points MFP will contain only one feature descriptor, although there may be more than one feature descriptor for each of those associated with each image in which the feature is present.

[0111] Alternatively, the step of determining a plurality of distinguishable features DF from the work site environment WE as a plurality of map feature points MFP further comprises a step in which the further map feature points MFP are determined at least in part from data retrieved from a ground movement information model, the ground movement information model being based on at least one of a geospatial information system (GIS), building information modelling (BIM), infrastructure or substructure building information modelling (I-BIM), civil information modelling (CIM) or a SmartCity Platform.

[0112] Alternatively, the plurality of map feature point MFPs are determined according to either the first embodiment for providing a map feature point MFP for creating a feature point map disclosed above, or the second embodiment for providing a map feature point MFP for creating a feature point map disclosed above, and further are at least partially determined from data retrieved from a ground movement information model, the ground movement information model being based on at least one of a geospatial information system (GIS), building information modeling (BIM), infrastructure or substructure building information modeling (I-BIM), civil information modeling (CIM), or a SmartCity Platform.

[0113] FIG. 8 shows a schematic disclosure of some ground movement or earthwork information models.

[0114] Fourth embodiment for providing map feature points for creating a feature point map Disclosed herein is a fourth embodiment for determining a plurality of map feature points MFP for creating a feature point map, i.e., a fourth embodiment for determining a plurality of distinguishable features DF as a plurality of map feature points MFP from a work site environment WE, the plurality of map feature points MFP having at least identification information and location data, the identification information including at least one unique identifier such as a feature descriptor for the map feature point MFP, and the location data including at least a three-dimensional location of the map feature point MFP in the work site 14.

[0115] According to this fourth embodiment, the map feature points MFP for creating the feature point map are determined as a combined set of map feature points MFP according to the method for providing map features for creating the feature point map disclosed in at least two of the first embodiment, the second embodiment or the third embodiment.

[0116] Referring again to Figures 2a and 2b and the associated description, Figures 7a, 7b and 7c, in view of Figures 2a and 2b, illustrate schematic diagrams of an embodiment for determining the location and orientation of a machine 1 in a work site 14. Given the availability of a pre-created feature point map including the map feature points MFP described above, the location and orientation of a machine 1 moving through the work site 14 can be determined.

[0117] To achieve this, the method comprises taking two-dimensional positioning images (PIM) substantially continuously by each of the at least one positioning camera PCA while the machine 1 is moving, at very short time intervals, such as between 10 ms and 100 ms, with an average time interval of about 50 ms for each positioning camera PCA. The time interval for taking the positioning images PIM can be extended if the machine 1 stops at the work site 14 without performing any operational action to realize a work phase.

[0118] At the same time as the two-dimensional positioning images PIM are taken, distinguishable features DF are detected from the two-dimensional positioning images PIM as navigation feature points (NFP), and two-dimensional image coordinates, i.e. image pixel coordinates, are determined for each navigation feature point NFP in each individual positioning image PIM, each of which has at least identification information. Thus, the content of these two-dimensional positioning images PIM is somewhat similar to the content of the work site environment image WIM shown in, for example, FIG. 5c, as can be seen in FIG. 7b.

[0119] At the same time as a two-dimensional positioning image PIM is taken, its content is analysed to detect distinguishable features DF from the positioning image PIM as navigation feature points NFP and to determine two-dimensional image coordinates for each navigation feature point NFP in the positioning image, and the navigation feature points NFP are matched with a number of map feature points MFP in a pre-created feature point map, whereby the identification information of the navigation feature points is matched with the identification information of at least one of the map feature points MFP or the detected non-matching feature points DFM-NM.

[0120] In response to matching the navigation feature points NFP and the map feature points MFP, - location data of at least one of the matched map feature points MFP or the detected non-matched feature points DFP-NM; - the 2D image coordinates of the corresponding matched navigation feature points NFP, and - the determined intrinsic parameter set and the determined location and orientation in the machine coordinate system MCS of each corresponding positioning camera PCA Based at least on the location and orientation of the machine 1 on the work site can be determined.

[0121] The location and orientation of machine 1 can be determined in at least two different ways, either of which can utilize sensors and / or groups of sensors present in machine 1, such as an inertial measurement unit (IMU), to provide additional robustness and / or accuracy.

[0122] According to a first embodiment or method herein, a PnP-solution (perspective-n-point problem) using three-dimensional information of the map feature points MFP that match the navigation feature points NFP can be applied. In this solution, a location and orientation of at least one positioning camera PCA in the work scene 14 is first determined based on the location data of each matched map feature point MFP and the two-dimensional image coordinates of each corresponding matched navigation feature point NFP. Thus, the input to the PnP-problem is the three-dimensional location data of the matched map feature points MFP and the two-dimensional image coordinates of the corresponding matched navigation feature points NFP, and the output of the PnP-problem is the location and orientation of at least one positioning camera PCA in the work scene 14. Based on the determined intrinsic parameter set and the determined location and orientation of each corresponding positioning camera PCA in the machine coordinate system MCS, the determined location and orientation of the at least one positioning camera PCA in the work scene 14 can be transformed to the location and orientation of the machine 1 in the work scene 14. PnP-issues in general are considered in more detail, for example, in "PnP-based Parallel Computing", IEEE Transactions on Information and Communications Technology Vol.

[0123] The localization results determined above can be improved via statistical prediction, for example by solving a nonlinear optimization problem for the variables. Such a nonlinear optimization problem originates from the well-known Bayesian statistics, which assumes that the measurement errors are normally distributed. The basic idea here is to assume that some values ​​for the state variables are solved (current state predictions), and compare feature measurements, i.e. image coordinates of the navigation feature points NFP from the camera and possible inertial measurements from the IMU and / or possible other measurements from other sensors or groups of sensors, against artificial / simulated measurements calculated by a camera projection model and possibly an IMU motion model and / or possibly models of other sensors or groups of sensors. Based on this comparison, the current state predictions are modified to reduce the comparison result. This adjustment process is possibly repeated until convergence, i.e. until no further significant improvement is obtained.

[0124] The information of the 3D locations of those features that match the map feature points in the feature point map can now be incorporated into the optimization problem as prior information for the 3D locations of those features that are included as state variables to be solved. In practice, this means that the function to be minimized in the optimization problem contains terms that compare current predictions for the 3D locations of the features that match the map feature points MFP against the location values ​​stored in the feature point map. Thus, as the state predictions are adjusted during optimization, these terms force the solution such that the 3D feature locations calculated from the camera measurements do not deviate from the 3D locations stored in the feature point map by at least more than allowed by the 3D location uncertainty information of the map feature points, thereby preventing accumulating fluctuations in the determined localization results.

[0125] According to a second embodiment or method herein, the location and orientation of the machine 1 are determined by one single image. In such an image, the location and orientation of which are precisely determined, the intrinsic parameters of the camera CA, PCA, which took the image are known, and the image contains at least five detected points with feature descriptors, such as ORB descriptors or ARUCO codes, etc., that allow the detection of the feature by the positioning camera PCA. Thus, in the second embodiment or method, the detected feature points DFP that are not matched as the same in at least two images can also be used in addition to the map feature points MFP, and if the unmatched detected feature points DFP contain at least a feature descriptor, the image identifier identifies the image in which the feature can be found and the two-dimensional image coordinates in which the feature is located in the image. This image identifier and the two-dimensional image coordinates may be present in the section for determining the two-dimensional image coordinates using the image identifier. These detected feature points DFP that are not matched in two or more images can be extremely useful, for example, in the edge regions of the work site or in the edge regions of the work area where the positioning of the machine is considered to have occurred.

[0126] According to a second embodiment or method herein, one of the at least one positioning cameras PCA detects at least five navigation feature points NFP that match at least one of the map feature points MFP or the detected non-matching feature points DFP-NM, and these points are present in one single image whose image location and orientation are known, i.e. whose camera CA, PCA location and orientation are known at the time when the image was taken by the camera CA, PCA. Such images are for example images of the accepted image group GIM, since once the location and orientation of each of the accepted image group GIM is determined, the location and orientation of each individual image in the image group GIM is also determined. In this embodiment, the location data of the map feature points MFP and the location data of the detected non-matching feature points DFP-NM include fields for determining two-dimensional image coordinates using image identifiers. The number of these location data fields for determining two-dimensional image coordinates using image identifiers in each map feature point MFP is at least one, and may be as many as there are images, whose location and orientation are determined by detecting the respective features. On the other hand, the number of these fields for determining the two-dimensional image coordinates using the image identifier for a non-matching feature point DFP-NM detected as the same point is only one. A non-matching feature point DFP-NM detected as the same point is shown, for example, in Figures 5b, 5c and 7c as a triangle with its tip pointing down and is designated by the reference character DFP-NM.

[0127] In this embodiment or method, at least one of the map feature points MFP or detected non-matching feature points DFP-NM is verified to have an identical image identifier in at least one field in the location data of the map feature points MFP or detected non-matching feature points DFP-NM present in these at least five identical images, and the image identifier allows the image location and orientation data of that image, i.e. the location and orientation data of the camera at the time the image was taken, to be determined. The two-dimensional image coordinates identify, for each image identifier field in at least one of the at least five map feature points MFP or detected non-matching feature points DFP-NM, the image in the image pixel in which the respective map feature point MFP or detected non-matching feature point DFP-NM is found. It should be noted that in this second solution or method, the location and orientation of the machine 1 can be determined by using only the detected non-matching feature points DFP-NM, if the two-dimensional positioning image PIM can detect at least five of them and the location data of those points verify points that have the same image identifier in the location data column and the image identifier teaches the image location and orientation, or by using, for example, either the leftmost or rightmost work site environment image WIM of Figure 5c.

[0128] Therefore, when at least five map feature points MFP or detected non-matching feature points DFP-NM are found in a single positioning image PIM, the location and orientation of such a positioning camera PCA can be determined by using these at least five navigation feature points NFP that match the map feature points MFP and / or detected non-matching feature points DFP-NM in a single image having a determined location and orientation of the single image determined, for example, at the time the location and orientation of the image group GIM was determined for each accepted image group GIM, or at the time an image was taken using, for example, a camera CA, PCA and its location and orientation was determined at the time the image was taken.

[0129] In this second embodiment or method, like the first embodiment or method, it is possible to improve accuracy by optimization.

[0130] The solution provides a robust and accurate determination of the location and orientation of the machine in the work site environment. The solution does not accumulate errors in determining the location and orientation of the machine. The solution creates a plurality of unique map feature points from a plurality of distinguishable features that are applied by the earthmoving machine as it moves and operates through the work site. The map feature points include identification information, which includes at least a unique identifier such as a feature descriptor and location data, including at least a three-dimensional location thereof in the work site environment, and / or two-dimensional image coordinates with an image identifier, and thus includes such location data fields identifying one or more images containing such detected map feature points, and the map feature points MFP are present in this image or images.

[0131] With reference to the above discussion related to the application of semantic information, the robustness of creating the feature point map can be further improved by semantic information obtained using distinguishable features. The semantic information can include, for example, information about the permanence of the object to which at least one distinguishable feature DF is related. Objects that, over time, may possibly completely disappear from the work site, or at least their position in the work site may possibly change, are preferably not applied as fixed points for map feature points in a feature point map intended for long-term use. Thus, for example, semantic information can be applied to purposely assign map feature points to objects that are robustly present in the work site environment.

[0132] Also, by calibrating the camera intended to be used for the application, the accuracy of creating the feature points and determining the location and orientation of the earthmoving machine on the work site can be improved as well. The camera intended to be used to detect distinguishable features that can be applied as map features for the feature point map can be calibrated at the beginning of the procedure of creating the feature point map. The camera intended to be used to detect distinguishable features that can be applied as navigation features during the movement or operation of the earthmoving machine on the work site can also be calibrated in real time during the maneuver. When calibrating a camera, its intrinsic parameters are predicted. The intrinsic parameters of a camera are related to its internal properties such as its image center, focal length, skew and lens distortion.

[0133] To increase the robustness of the determination of the machine location and orientation in real time during the machine movement and operation, e.g. robustness to changes in the camera model parameters due to shocks and vibrations that occur, the intrinsic camera parameters may be included as variables to be optimized in a nonlinear optimization problem constructed from data collected during the maneuver. This nonlinear optimization problem is similar to the nonlinear optimization problem described above, with the difference that here the intrinsic parameter set of the positioning camera PCA should be optimized / improved together with the other parameters. For that it is necessary to be able to observe the camera parameters from data, i.e. feature measurements obtained from different camera locations, and therefore such data must be selected that makes the camera intrinsic parameters observable. A measure of observability can be defined based on the uncertainty of the predicted parameters, such as the covariance matrix, given a particular data set collected during the maneuver. The particular data set may here be a particular segment of the camera path, including feature measurements, which are performed as the camera moves through this path. If the measure of observability is good enough, a parameter update is performed. Multiple data sets with good observability of the camera intrinsic parameters may also be included in the predictions. It is also possible to detect changes in the intrinsic camera parameters by comparing two different sets of predictions for these parameters, such as the predictions determined from the most recent data set with good observability and the predictions from many data sets collected during a longer time period, where a difference in such predictions indicates a change in the intrinsic camera parameters. This information can be used to determine when the parameters need to be re-predicted. This prediction process is performed in parallel with the localization process. If this parallel process successfully solves the parameters, the camera intrinsic parameters used in the localization process are replaced with the new parameters.

[0134] According to an embodiment, the step of determining the location and orientation of the earthwork machine 1 at the work site 14 is further based on a step of tracking navigation feature points NFP between successive positioning images PIM, the navigation feature points NFP tracked between successive positioning images PIM indicating changes in the location and orientation of the positioning camera PCA.

[0135] This embodiment provides a visual odometry procedure that can be applied for short-term determination of the location and orientation of the machine in the work site 14 when the navigation features NFP detected from the positioning image PIM cannot be matched with the map features MFP in the feature map for some reason. The reason for using visual odometry may be, for example, fog, snowfall, rainfall, extreme dry weather, frost or dust. This method makes the determination of the location and orientation of the machine more robust and can manage the determination over short periods when the navigation features NFP do not match the map features MFP well.

[0136] If matching of the navigation feature points NFP to the map feature points MFP in the feature point map cannot be performed within some short period of time, a simultaneous localization and map creation procedure can be initiated to create a temporary feature point map based on the resolved three-dimensional locations of the tracked features in the captured positioning images, and to localize the machine 1 within this temporary feature point map, in which the detected features are used as map features.

[0137] Fig. 9 shows a schematic representation of an embodiment relating to a visual odometry procedure. Fig. 9 shows the machine 1 in a position on the work site 14 when the preceding positioning image PIM is taken, in dotted lines, and in a continuous line in a position on the work site 14 when the positioning image PIM shown in Fig. 9 is taken. The features detected on the work site 14 by the positioning camera PCA are represented by filled cross symbols. In the positioning image PIM, these detected features are set at the locations indicated by diamond symbols. The dotted lines in the positioning image PIM represent the movement of the position of the detected features in the positioning image between the preceding positioning image and the current positioning image, reflecting the change in position and orientation of the positioning camera PCA taking the positioning image PIM.

[0138] In the simultaneous localization and map building procedure, features detected in the latest positioning image are attempted to be matched not only against the actual feature map but also against the temporary feature map. This method assumes that the temporary feature map created has good accuracy, but can maintain good localization accuracy even when feature matching against the actual feature map over a long period of time is unsuccessful.

[0139] In order to create a temporary feature point map with good accuracy, the temporary feature point map can be updated after successful implementation of the loop detection and loop closure functionality. Loop detection means that the area in the work site 14 where the machine was previously located, i.e. the area of ​​the work site that is visible in one or more previous positioning images PIM, can be recognized based on a comparison of the latest positioning image with the positioning image taken at the previous time. A characteristic value, for example a so-called Bag-of-Words, can be calculated from the positioning images for comparison. When a loop is detected, in the case of localization by using only odometry, the accumulated fluctuations, i.e. the localization errors, during the search phase can be corrected using the loop closure functionality, which takes into account the constraints of the search path that form the loop, i.e. the loop constraints, and uses these constraints to reoptimize the path and the local map created during the search. This process is performed in parallel with the real-time localization process. Once the reoptimization is successful, the temporary feature point map can be updated using the new map points.

[0140] According to an embodiment, the location and orientation of the earthwork machine 1 is further determined in an earthwork information model, the earthwork information being based on at least one of a Geospatial Information System (GIS), a Building Information Modeling (BIM), an Infrastructure or Substructure Building Information Modeling (I-BIM), a Civil Information Model (CIM) or a SmartCity Platform. According to this embodiment, the earthwork information model in use for the work site 14 can be updated substantially in real time to include information about the location and orientation of the machine 1 at the work site 14. This information can be used, for example, as a further possible measure for detecting possible crossing of safety boundaries set around other machines or people operating at the work site 14 and for controlling the machine 1 to stop if necessary.

[0141] According to an embodiment, the tools of the earthworking machine 1 are controlled to create a structure described in the earthworking information model. According to this embodiment, the machine 1 can receive e.g. specific work instructions from the earthworking information model, even in the form of specific control actions that are controlled and executed by the control unit 11 in order to complete the intended planned structure described in the earthworking information model.

[0142] According to the embodiment, as-built data is generated for the earthwork information model to follow. According to this embodiment, the earthwork information model can be supplemented with as-built data describing the progress of the work site 14. The as-built data relating to the progress of the work site 14 can be generated within the earthwork machine 1 or can be generated by the earthwork machine 1 as it moves through the work site 14 or as it performs work tasks to complete the planned structure of the work site 14. Thus, the machine 1 can collect as-built data relating to the progress of the work site 14 in response to work tasks performed by the machine 1, as well as in response to work tasks performed by other machines at the work site 14.

[0143] According to an embodiment, the as-built data is transmitted to a work site information management system or to at least one of the machines operating at the work site 14. The as-built data may be transmitted in response to completion of a planned structure at the work site 14, in response to completion of a sub-structure forming part of the planned structure, or in response to completion of a particular individual operation performed to realize the sub-structure.

[0144] It is obvious to those skilled in the art that as technology advances, the concept of the present invention can be realized in various ways. The present invention and its embodiments are not limited to the examples described above, but can be modified within the scope of the claims.

Claims

1. 1. A method for determining a location and orientation of an earthmoving machine in a work site coordinate system, the method comprising: Creating a feature point map, determining a plurality of distinguishable features as a plurality of map feature points from the work site environment by taking a plurality of two-dimensional work site environment images with at least one camera having a determined intrinsic parameter set, the intrinsic parameter set defining a formation of individual image pixels from a real-world view for the images; from each of the plurality of two-dimensional worksite environment images; Distinguishing features for associating each distinct feature with an identity and image coordinates of each two-dimensional worksite environment image as a detected feature point; or Distinguishable features as reference feature points, the reference feature points being preset at the work site, and associated with identification information and location data in the work site coordinate system to further associate each of the reference feature points with image coordinates of each two-dimensional work site environment image as a detected reference feature point. detecting at least one of Creating at least one image group, combining work site environment images together by having at least five of the detected feature points or at least one of the detected reference feature points matched as the same point in the work site environment images to be combined; receiving at least two worksite environment images and respective image groups including at least three detected reference feature points that are matched as identical points; determining, for each accepted group of images, a location and orientation of the group of images and identification and location data for detected feature points that are matched as identical points, by utilizing the identification and location data of the at least three detected reference feature points included therein, determining the detected feature points that are matched as the same point and the detected reference feature points that are matched as the same point as map feature points for the feature point map, the map feature points of the feature point map having at least identification information and location data, the identification information including at least a unique identifier that identifies the distinguishable feature, and the location data including a three dimensional location in the work site coordinate system; The steps to create and establishing at least one positioning camera above said earth moving machine, said at least one positioning camera having a determined set of intrinsic parameters and a determined location and orientation in a machine coordinate system; providing the feature points map to the earthmoving machine; capturing two-dimensional positioning images with at least one positioning camera; detecting distinguishable features from the two-dimensional positioning image as navigation feature points and determining two-dimensional image coordinates and identification information for each navigation feature point; matching the identification information of the navigation feature with the identification information of the map feature in the feature map; Including, Determining the location and orientation of the earthmoving machine in the worksite coordinate system includes at least said location data of matched map features; the two-dimensional image coordinates of the corresponding matched navigation features; and the determined intrinsic parameter set for each of the at least one positioning camera, and the determined location and orientation in the machine coordinate system of each corresponding positioning camera. Based on,method.

2. determining a plurality of distinct features from the work site environment as a plurality of map feature points; capturing a plurality of two-dimensional work site environment images with at least one camera, the at least one camera comprising: a determined set of intrinsic parameters; and the determined location in said work site coordinate system; and detecting distinct features from each of the plurality of two-dimensional worksite environment images and determining the distinct features as detected feature points using image coordinates of the respective two-dimensional worksite environment images; creating at least one image group having detected feature points that match as same points in the worksite environment image; determining, for each of the group of images, identification and location data for the detected feature points that are matched as identical by utilizing the location in the worksite coordinate system of the at least one camera having a respective respective two-dimensional worksite environment image; Further comprising: The method of claim 1 , wherein the detected feature points that match the same determined identity and location data are determined as a plurality of map feature points.

3. 2. The method of claim 1, wherein the step of determining a plurality of distinguishable features from the work site environment as a plurality of map feature points further includes other map feature points determined at least in part from data retrieved from a ground movement information model, the ground movement information model being based on at least one of a geospatial information system, building information modeling, infrastructure or substructure building information modeling, civil information modeling, or the SmartCity Platform.

4. For each accepted image group, the step of determining the location and orientation of the image group and the identification and location data for the detected feature points that are matched as the same further comprises the step of determining a static rating for the detected feature points that are matched as the same by utilizing the identification and location data of the at least three detected reference feature points included therein; The method of claim 1 , wherein at least one of the detected feature points that match the determined identification information, location data and the static rating as being the same, or the detected reference feature points associated with the identification information and location data, are determined as the plurality of map feature points.

5. determining, for each group of images, identification information and location data for the detected feature points that are matched as identical further comprises determining static ratings for the detected feature points that are matched as identical by utilizing the location in the work site coordinate system of the at least one camera having a respective respective two-dimensional work site environment image; The method of claim 2 , wherein the detected feature points that match the same determined identity information, location data, and static rating are determined as a plurality of map feature points.

6. 6. The method of claim 4 or 5, wherein the method further comprises the step of determining an overall rating, the overall rating comprising at least one of a static rating or a dynamic rating, the dynamic rating being determined for each of the plurality of determined map feature points with respect to an individual positioning camera, taking into account a previously determined location and orientation of each of the at least one positioning camera individually.

7. The method of claim 6 , wherein the overall rating is at least one of the static rating or the dynamic rating, and the overall rating has at least two tiers.

8. 8. The method of claim 7, wherein at least the map feature point having the lowest stratum overall rating is discarded if the amount of map feature points is greater than a threshold, the threshold being at least one of manually or automatically determined.

9. 6. The method of claim 1, wherein determining the location and orientation of the earthmoving machine at the work site is further based on tracking navigational feature points between successive positioning images, the navigational feature points tracked between the successive positioning images being indicative of changes in location and orientation of the positioning camera.

10. 6. The method of claim 1, further comprising determining the location and orientation of the earthmoving machine in an earthmoving information model, the earthmoving information being based on at least one of a geospatial information system, building information modeling, infrastructure or substructure building information modeling, civil information modeling and the SmartCity Platform.

11. The method of claim 10 , further comprising controlling a tool of the earthmoving machine to create a structure described in the earthworking information model.

12. The method of claim 10 , further comprising the step of creating as-built data for the earthworks information model to track.

13. The method of claim 12 , further comprising transmitting the as-built data to at least one of a work site information management system or a machine operating at the work site.

14. 6. A method according to claim 1, wherein the intrinsic parameter set is updated with at least one data set collected during a maneuver, the data set being created by a particular segment of a camera path including feature measurements, the feature measurements being taken as the camera moves through the camera path.

Images