Method for determining situational awareness in a job site

Abstract

A method for determining situational awareness in a work site (13) includes providing at least one environmental modeling device (EM) on or outside of a machine; providing at least one tracking device (TA) on or outside of the machine; acquiring data by the at least one tracking device (TA) and acquiring data by the at least one environmental modeling device. In addition, the method includes receiving data related to the at least one tracking device (TA) and data related to the at least one environmental modeling device (EM) by at least one position determining unit (PDU), and determining, by the at least one position determining unit (PDU), a position and orientation of the machine in the work site (13) based at least in part on the received data.

Description

Technical Field

[0001] The present invention relates to a method for determining situational awareness in a work site. Background Art

[0002] Different types of work machines may be used at different earthmoving work sites or construction sites, for example, for moving soil or rock material from one location to another, or for lifting or lowering material for construction. Examples of such work sites include, for example, underground structure construction work sites for buildings or house construction work sites and road construction work sites, which form some examples of typical work sites. Work machines like that are, for example, excavators and mobile cranes.

[0003] A working machine and a working tool of the working machine should be able to be positioned very accurately at the work site in order to correctly carry out the designed operation. Information about the exact position of the working machine and its tool can be displayed to the operator of the working machine so that the operator uses this information when controlling the tool and the machine. The exact position information of the machine and its tool is particularly important when it is used in a semi-automatic or fully automatic working machine, i.e. the working machine is operated (at least for a period of time) without continuous control by the operator of the machine, so that the operator of said machine does not immediately correct possible misalignments of the machine or its tool.

[0004] Typically, automatic positioning of the machine may be based on, for example, a satellite positioning system GNSS (Global Navigation Satellite System), such as GPS (US), GLONASS (RU), Galileo (EU) or Compass (CN). Alternatively, positioning of the work machine may be provided by a total station positioned at the work site.

[0005] However, there may not be any or sufficiently accurate satellite positioning systems available at every job site, or the available positioning systems may not be sophisticated enough to determine the position and orientation of the machinery with high accuracy. Retrofitting older machinery with high-precision systems may not be very cost-effective. In addition, setting up a total station positioning system at a job site may be laborious, especially if the total station positioning system is removed from the job site every day or multiple times a day.

[0006] In addition to accurate positioning of the work machine at the work site, the condition or status of the work site should also be understood, providing information describing situational awareness at the work site, which will facilitate efficient operation of the work machine and progress at the work site.

[0007] Therefore, there is a need for a simple positioning solution that also improves situational awareness on the job site. Summary of the invention

[0008] It is an object of the present invention to provide a new method for determining situational awareness in a work site.

[0009] The invention is characterized by the features of the independent claims.

[0010] In the present invention, in order to determine situational awareness in a work site, determination of the position and orientation of the machine in the work site is combined with providing knowledge of the condition or state of the work site.

[0011] The positioning of the work machine in the work site and the acquisition of the conditions or status of the work site provide information that depicts situational awareness in the work site, promoting the efficient operation of the work machine and the progress of the work site. This allows the control of the work machine to also take into account other work machines and alternate situations or unexpected events, which occur not only during the performance of the work task currently being performed, but also in future work tasks to be performed.

[0012] Some embodiments of the invention are disclosed in the dependent claims.

[0013] According to an embodiment of a method for determining situational awareness in a work site, the method includes: setting at least one environmental modeling device as at least one of the following: on a machine or outside a machine; setting at least one tracking device as at least one of the following: on a machine or outside a machine; acquiring data by at least one tracking device; acquiring data by at least one environmental modeling device; receiving data related to at least one tracking device and data related to at least one environmental modeling device by at least one position determination unit; and determining, by at least one position determination unit, a position and orientation of the machine in the work site based at least in part on the received data.

[0014] According to an embodiment of the method, the method further comprises determining, by the at least one position determination unit at least partly based on the received data, at least one of: a driving direction or an alternative driving direction of the machine in the work site.

[0015] According to an embodiment of the method, the method further comprises determining at least one of: a level of accuracy or validity of the determined position and orientation of the machine in the work site.

[0016] According to an embodiment of the method, the data related to at least one tracking device includes at least one of the following: the position of a tracking marker point relative to the tracking device, the position of a tracking reference point relative to the tracking device, the inclination of the tracking device, the heading of the tracking device, the stability of the tracking device, the position and orientation of the tracking device in at least one of a machine coordinate system, a work site coordinate system or a world coordinate system, or at least one of the accuracy level or validity of at least one of the previous ones.

[0017] According to an embodiment of the method, the data related to at least one environmental modeling device is at least one of the following: spatial data, the position of a tracking marker point relative to the environmental modeling device, the position of a tracking reference point relative to the environmental modeling device, the inclination of the environmental modeling device, the heading of the environmental modeling device, the stability of the environmental modeling device, the position and orientation of the environmental modeling device in at least one of a machine coordinate system, a work site coordinate system or a world coordinate system, or at least one of the accuracy level or validity of at least one of the previous ones.

[0018] According to an embodiment of the method, the spatial data comprises at least one of: graphical data, point cloud data, or data implicitly or explicitly referencing a location relative to at least one of: a work site or the earth.

[0019] According to an embodiment of the method, the method also includes: receiving, by at least one environmental modeling unit, an indication of material delivery, a material delivery base determined by first data related to at least one environmental modeling device covering the area where the material is to be placed, and material delivery completion determined by second data related to at least one environmental modeling device covering the area where the material is to be placed; and at least one environmental modeling unit at least partially saving the indication of material delivery, the material delivery base, and the data on material delivery completion as delivered materials.

[0020] According to an embodiment of the method, the method also includes: receiving, by at least one environmental modeling unit, one or more indications related to at least one of the working periods or working stages of the corresponding area, and data related to at least one environmental modeling device; and wherein, by considering one or more indications, the geo-referenced spatial data of the corresponding area is derived from the data related to at least one environmental modeling device, and the geo-referenced spatial data is at least partially saved.

[0021] According to an embodiment of the method, the step of at least partially saving geo-referenced spatial data further comprises determining an area for which geo-referenced spatial data is to be saved based at least partially on data received from at least one environment modeling device, and saving the geo-referenced spatial data of the area determined to be saved.

[0022] According to an embodiment of the method, the step of determining an area whose geo-reference spatial data is to be saved based at least in part on data received from at least one environmental modeling device includes: detecting an area where the geo-reference spatial data of at least one environmental modeling device is unobstructed and treating it as an unobstructed area, and the method also includes: comparing the current accuracy of the determined position and orientation of the machine at the work site with the accuracy of the determined position and orientation of the machine at the work site during the time of the previously saved geo-reference spatial data, and whether the current accuracy is almost the same, updating the geo-reference spatial data saved in the unobstructed area.

[0023] According to an embodiment of the method, the method also includes: parsing data related to at least one of the following by at least one position determination unit that determines the position and orientation of the machine in the work site: a tracking device, an environmental modeling device, an object or another machine; and transmitting the parsed data as at least one of the following: part of the data related to the corresponding tracking device, environmental modeling device, object or another machine, or data that can be received by at least one position determination unit.

[0024] According to an embodiment of the method, the method further comprises initializing the tracking device by determining a position and orientation of the tracking device in at least one of a machine coordinate system or a work site coordinate system if the tracking device is disposed on at least one of the machine or another machine, and determining a position and orientation of the tracking device in a work site coordinate system if the tracking device is disposed outside any machine.

[0025] According to an embodiment of the method, the method also includes initializing the environmental modeling device by: if the environmental modeling device is set on at least one of the machine or another machine, determining the position and orientation of the environmental modeling device in at least one of the machine coordinate system or the work site coordinate system, and if the environmental modeling device is set outside any machine, determining the position and orientation of the environmental modeling device in the work site coordinate system.

[0026] According to an embodiment of the method, the determined situational awareness is: at least one of spatial data, geo-referenced spatial data, regional work period data, regional work stage data, completion data, at least one of the position, orientation, driving direction or alternative driving direction of any machine in the work site, at least one of the position, orientation or driving direction or alternative driving direction of at least one of the machines, obstacles or objects that are stationary or moving, to be avoided or of interest, or at least one of the surrounding environment.

[0027] According to an embodiment of the method, at least one tracking device tracks the position of at least one of the following relative to the tracking device: at least one reference point in the work site, at least one marking point attached to the machine, or any other trackable marker attached to at least one of the machine, obstacles or objects; and at least one environmental modeling device tracks the position of at least one of the following relative to at least one environmental modeling device: at least one reference point in the work site, at least one marking point attached to the machine, any other trackable marker attached to at least one of the machine, obstacles or objects, or spatial data related to the work site.

[0028] According to an embodiment of the method, the method further comprises: determining a minimum level of accuracy of the determined position and orientation of the machine at the work site; determining a threshold level above the minimum level of accuracy, and wherein, if the level of accuracy is below the threshold level and if the ongoing work task can be performed without moving the lower carriage, a control device for moving the lower carriage of the machine is disabled.

[0029] According to an embodiment of the method, data related to at least one of at least one tracking device or at least one environmental modeling device includes at least one of the following: data acquired by the corresponding device, data from a sensor mounted on the corresponding device, data from a sensor mounted on an attachment point of the device, data parsed by at least one of any position determination unit or any device by tracking the corresponding device or as a result of any calculation related to the corresponding device, or the accuracy level or validity of at least one of the previous.

[0030] According to an embodiment of the method, if at least one tracking device is arranged on the work site and if it includes a tracking device for tracking the position of the tracking device using one or more GNSS antennas, the tracking device also includes at least one of the following as the tracking device: a camera, a stereo camera, a lidar, a radar or a speedometer.

[0031] According to an embodiment of the method, the position and orientation of the machine in the work site is determined by at least one position determination unit additionally at least partially based on data received from one or more sensors mounted on at least one of the machine or the other machine, wherein the sensors include at least one of the following: the position, orientation, inclination, heading or travel distance of at least one of the machine or the other machine.

[0032] According to an embodiment of the method, the machine is an excavator, and the position and orientation of the machine in the work site is determined by at least one position determination unit additionally at least partially based on data received from one or more sensors mounted on an upper bracket of at least one of the machine or the other machine, wherein the sensors include at least one of the following: the position, orientation, inclination or heading of the upper bracket of at least one of the machine or the other machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Hereinafter, the present invention will be described in more detail by way of preferred embodiments with reference to the accompanying drawings, in which

[0034] Figure 1 A side view of an excavator is schematically shown;

[0035] Figure 2 A top view of a work site is schematically shown;

[0036] Figure 3 Some components of a positioning system for determining the position and orientation of a machine at a work site are schematically shown;

[0037] Figure 4 An embodiment of a method for determining the position and orientation of a machine in a work site is schematically illustrated;

[0038] Figure 5 Schematically illustrates a reference marker having a number of reference points to be arranged in a work site for determining the position and orientation of a machine in the work site;

[0039] Figure 6 A marking having a plurality of marking points is schematically shown, the marking points being arranged on a machine for determining the position and orientation of the machine in a work site;

[0040] Figure 7 An embodiment of a tracking device is schematically shown;

[0041] Figure 8 schematically illustrates sensors that may be arranged in a machine and / or tracking device;

[0042] Fig. 9 Some possible tracking states of the tracking device are schematically shown;

[0043] Fig.10 schematically shows a top view of a second work site;

[0044] Fig.11 An embodiment of another method for determining the position and orientation of a machine in a work site is schematically illustrated;

[0045] Fig.12 schematically shows a top view of a third work site;

[0046] Fig.13 schematically shows a top view of a fourth work site;

[0047] Fig.14 Some components of a positioning system for determining the position and orientation of a machine in a work site and for situational awareness in a work site are schematically shown;

[0048] Fig.15 An embodiment of another method for determining the position and orientation of a machine in a work site is schematically illustrated;

[0049] Fig.16 schematically illustrating data associated with at least one tracking device;

[0050] Fig.17 schematically illustrating data associated with at least one environmental modeling device; and

[0051] Fig.18 Data depicting situational awareness in a work site is schematically illustrated.

[0052] For the sake of clarity, the accompanying drawings show some embodiments of the present invention in a simplified manner. In the accompanying drawings, similar reference numerals represent similar elements. DETAILED DESCRIPTION

[0053] Figure 1is a schematic side view of an excavator 1 at a work site 13, in which the excavator 1 is to be operated. The excavator 1 is an example of a work machine that can be used in conjunction with the disclosed method and positioning system and method for determining situational awareness in the work site 13. The work site 13 includes at least one area or space where active work is to be performed. Depending on the nature or characteristics of the work site 13, the work site 13 may include, in addition to the area or space where active work is to be performed, one or more areas or spaces, typically surrounding areas or spaces, which may affect operations at the area or space where active work is to be performed, and / or operations performed at the area or space where active work is to be performed may affect the areas or spaces. In particular, in urban environments, the work machine may need to work outside the actual or officially determined work site, and therefore, in the present description, the work site 13 may extend several meters beyond the area and / or space representing the actual or officially determined work site, and therefore obstacles and pedestrians near the actual or officially determined work site may need to be tracked or detected. Furthermore, the work site 13 may also include at least some portions of the surroundings of the work site 13 that are not explicitly indicated, for example, by corresponding visual information signs that are part of the work site 13 .

[0054] The excavator 1 comprises a movable bracket 2, which comprises: a lower bracket 2a (i.e., lower bracket 2a) and an upper bracket 2b. The lower bracket 2a comprises tracks, but wheels may be provided instead. The upper bracket 2b is connected to the lower bracket 2a via a rotation axis 3 of the upper bracket 2b. The upper bracket 2b can rotate relative to the lower bracket 2a around a rotation axis 4, as schematically shown by arrow R. The rotation axis 4 coincides with the central axis of the rotation axis 3.

[0055] The excavator 1 further comprises a boom 5 connected to the upper bracket 2b, whereby the boom 5 is arranged to rotate together with the upper bracket 2b. The boom 5 may comprise at least a first boom component 5a. The boom 5 may also comprise other boom components, for example, a second boom component 5b. The boom 5 may be raised and lowered relative to the upper bracket 2b, as schematically shown by arrow L.

[0056] The second boom part 5b may be connected to the first boom part 5a by a joint 6, allowing the second boom part 5b to rotate around the first boom part 5a, as schematically shown by arrow T6. At the distal end of the second boom part 5b, there is a working tool, in this case a bucket 7, and between the bucket 7 and the second boom part 5b, there may be a joint 8, allowing the bucket 7 to rotate around the second boom part 5b, as schematically shown by arrow T8. For example, in connection with the joint 8, there may also be a joint or mechanism that allows the bucket to tilt sideways.

[0057] On the carriage 2 there may be a control cabin 9 for an operator 10 of the excavator 1. The control cabin 9 may for example be provided with a moving arrangement allowing the vertical position of the control cabin 9 relative to the carriage 2 to be adjusted.

[0058] The excavator 1 further comprises at least one control unit 11 configured to control the operation of the excavator 1 , for example the operation of the carriage 2 , the boom 5 and the bucket 7 , in response to received control actions.

[0059] If the excavator 1 is intended to be able to utilize a satellite-based positioning system GNSS (Global Navigation Satellite System), the excavator 1 may also comprise satellite receiving means such as an antenna 12. The antenna 12 may, for example, be placed on the upper bracket 2b.

[0060] Figure 2 Schematically, a top view of a work site 13 is shown, wherein the excavator 1 is about to be operated. Figure 2 In the example of , at a work site 13 there is an excavator 1 and some equipment of a positioning system PS for determining the position and orientation of the excavator 1 (or alternatively, some other machine) in a work site coordinate system WCS. Furthermore, the control system of the excavator 1 may include its own mechanical coordinate system MCS, whereby the mechanical coordinate system MCS may be fixed to the machine with marking points MP arranged in the machine, so that the positioning provided by the positioning system PS allows the mechanical coordinate system MCS to be identified relative to the work site coordinate system WCS. Figure 1 The work site coordinate system WCS and the mechanical coordinate system MCS are schematically shown in FIG. Figure 3 Furthermore, an embodiment of a positioning system PS is schematically shown with some additional equipment related to the positioning system PS.

[0061] The positioning system PS comprises at least one reference marker RM, i.e. one or more reference markers RM arranged in the work site 13. The reference marker RM arranged in the work site 13 may be, for example, an aruco marker, a QR code, a luminous marker, a reflective marker, a prism, etc. Each reference marker RM provides at least one reference point RP, i.e. one or more reference points RP, whereby there is at least one reference point RP in the work site 13, which is determined in the work site coordinate system WCS for determining the position and orientation of the machine. For the sake of clarity, in this specification, determining and / or defining and / or calculating the orientation of the machine and / or equipment in the work site and / or coordinate system means: determining the three required angles, e.g., roll, pitch and yaw, to accurately determine the position of the machine and / or equipment in the work site and / or coordinate system. Therefore, when the position of at least one point and / or location of a machine and / or equipment (which point and / or location is known in its own coordinate system) is determined or can be determined, and the orientation of the machine and / or the equipment is determined or can be determined, the machine and / or the equipment can be accurately fixed and / or placed and / or set and / or positioned to the work site or coordinate system.

[0062] exist Figure 2 In the example of , three reference markers RM are schematically shown, i.e., a first reference marker RM1, a second reference marker RM2, and a third reference marker RM3, which are set or arranged in the work site 13. Each of the reference markers RM1, RM2, RM3 includes a reference point RP, i.e., the first reference marker RM1 includes a first reference point RP1, the second reference marker RM2 includes a second reference point RP2, and the third reference marker RM3 includes a third reference point RP3. The specific reference point RP can be provided by a specific point in an Aluco marker or by a specific light-emitting device in a light-emitting marker, for example. In Figure 5 In the example of FIG, a reference marker RM having three reference points RP (i.e., a first reference point RP1, a second reference point RP2, and a third reference point RP3) is schematically shown. Each reference point RP is identifiable, and the position of the reference point RP is determined in the work site coordinate system WCS. Therefore, with the identification data of the reference point RP, the position of the reference point RP in the work site coordinate system WCS can be determined.

[0063] The positioning system PS also includes at least one marker MA, that is, one or more markers MA, arranged on the machine (i.e., in the excavator 1). The marker MA arranged on the machine can be, for example, an Aluco marker, a QR code, a luminous marker such as an optical tracker, a reflective marker, etc. Each marker MA provides at least one marker point MP, that is, one or more marker points MP, so that there is at least one marker point MP on the machine for determining the position and orientation of the machine. Each marker point MP of each machine is identifiable, and the position of the marker point MP is determined in the machine coordinate system MCS, so that the positioning system PS that locates and identifies the marker point MP can locate the machine coordinate system MCS, that is, the machine. Therefore, with the identification data of the marker point MP, the position of the marker point MP in the machine coordinate system MCS can be determined.

[0064] exist Figure 1 In the example of FIG. 1 , two markers MA arranged on the machine are schematically shown, namely, a first marker MA1 and a second marker MA2. Each marker MA1, MA2 includes a marking point MP, namely, the first marker MA1 includes a first marking point MP1, and the second marker MA2 includes a second marking point MP2. For example, a specific marking point MP can be provided by a specific point in an Aluco marker or by a specific light-emitting device in a light-emitting marker. Figure 6 In the example of FIG. 1 , a marker MA having three marking points MP (ie, a first marking point MP1 , a second marking point MP2 , and a third marking point MP3 ) is schematically shown.

[0065] The positioning system PS further comprises at least one tracking device TA, i.e., more than one tracking device TA arranged in the work site. The tracking device TA tracks or monitors reference points RP and marker points MP, in particular identification data and their positions in the work site 13 relative to the tracking device TA. Based on initialization of tracking, the tracking device TA tracks the position of at least one identified reference point RP in the work site 13 and the position of at least one identified marker point MP in the machine. Thus, by using the identification data of at least one reference point RP, the positioning system PS is able to locate the reference point RP in the work site coordinate system WCS; by using the identification data of at least one marker point MP, the positioning system PS is able to locate the marker point MP in the machine coordinate system MCS; and, after the positioning system PS has tracked the positions of at least one identified reference point RP and at least one identified marker point MP relative thereto, the positioning system PS is able to determine the position and orientation of the machine coordinate system MCS (i.e., the machine) in the work site coordinate system WCS. The tracking device TA comprises at least one tracking device TD, i.e., one or more tracking devices TD, to provide visual communication between the tracking device TA and the reference points RP and the marker points MP. Furthermore, the tracking device TA comprises means for receiving and / or sending information, for example an input-output unit.

[0066] Tracking or monitoring is performed or carried out by visual communication between the tracking device TA and the corresponding reference points RP and marker points MP. Figure 2 In the example of , there are two tracking devices TA arranged in the work site 13, namely, a first tracking device TA1 and a second tracking device TA2. The first tracking device TA1 includes three tracking devices, namely, a first tracking device TD1, a second tracking device TD2 and a third tracking device TD3, to provide a first visual connection TD1_RP1 between the first tracking device TD1 and the first reference point RP1, a second visual connection TD2_RP2 between the second tracking device TD2 and the second reference point RP2, and a third visual connection TD3_RP3 between the third tracking device TD3 and the third reference point RP3 at a certain moment. In addition, the second tracking device TD2 is arranged to provide a fourth visual connection TD2_RM2 between the second tracking device TD2 and the second marking point MP2. The second tracking device TA2 includes two tracking devices, namely, a fourth tracking device TD4 and a fifth tracking device TD5, the fourth tracking device TD4 providing a fifth visual connection TD4_MP1 between the fourth tracking device TD4 and the first marking point MP1, and the fifth tracking device TD5 providing a sixth visual connection TD5_MP2 between the fifth tracking device TD5 and the second marking point MP2. Furthermore, the fifth tracking device TD5 is arranged to provide a seventh visual connection TD5_RP2 between the fifth tracking device TD5 and the second reference point RP2.

[0067] exist Figure 2In the embodiment, two tracking devices are shown, namely, a first tracking device TA1 and a second tracking device TA2, wherein the first tracking device TA1 is provided with three tracking devices TD1, TD2, TD3 to track the positions of reference points RP1, RP2, RP3 and the position of the second marker MP2 at a certain moment, and the second tracking device TA2 is provided with two tracking devices TD4, TD5 to track the positions of markers MP1, MP2 and the position of the second reference point RP2 at a certain moment. However, each of the tracking devices TA1, TA2 may include only a single tracking device to track the positions of the corresponding reference points RP1, RP2, RP3 and the positions of markers MP1, MP2. In addition, the positioning system may include only a single tracking device, which has one or more tracking devices TD to track the positions of reference points RP1, RP2, RP3 and markers MP1, MP2. In addition, if two or more tracking devices TA (e.g., TA1, TA2) are in known orientations and positions relative to each other, then the arrangement of all tracking devices TD (e.g., TD1, TD2, TD3, TD4, TD5) corresponding to having one tracking device TA and two or more tracking devices TA (e.g., TA1, TA2) is such that, when TA1 only tries to track the reference point RP and TA2 only tries to track the marker point MP, the combination of all acquired data can determine the position and orientation of the machine in the work site 13.

[0068] The tracking device TD1, TD2, TD3, TD4, TD5 is a device capable of establishing or providing a visual connection between the tracking apparatus TA and at least one reference point RP and / or at least one marker point MP. According to one embodiment, the tracking device is a camera, a stereo camera, a lidar or a velocimeter.

[0069] The camera and / or stereo camera may be equipped with a lens or objective having such a focus that the camera and / or stereo camera can establish an accurate visual connection with an object that remains substantially near the camera and / or stereo camera or is relatively far away from the camera and / or stereo camera (e.g., an object that is substantially much farther away from the camera and / or stereo camera than the machinery in the work site 13). Thus, the camera and / or stereo camera may be able to provide a zooming effect to an object that is substantially far away from the tracking device TA.

[0070] According to an embodiment of the tracking apparatus TA, the tracking apparatus TA comprises a camera or some other tracking means TD and a base B of the camera or some other tracking means, whereby the tracking apparatus TA can be set at a specific fixed position at the ground in the work site 13 . Figure 7 Such a tracking device TA is schematically shown on the left.

[0071] In order to obtain high accuracy using one camera or some other tracking device TD, the tracking device TA, the tracked reference points RP and the marker points MP should be farther from each other and / or they can be not too far from the tracking device TA. Using one camera, preferably, there should be at least four reference points RP forming a plane and at least four marker points MP also forming a plane. Using two, three or more tracking devices TD in the tracking device TA, the distance between the tracking device TA and the reference points RP and the marker points MP can be longer, and the number of tracked reference points RP and marker points MP can be less. In other words, the more tracking devices TD there are, the more reference points RP are tracked, the more marker points MP are tracked, and the smaller the distance between the tracking device TA and the tracked reference points RP and the tracked marker points MP, the higher the accuracy. Remember that the positions of the tracked reference points RP compared to each other affect the accuracy, and the positions of the tracked marker points MP compared to each other affect the accuracy. Preferably, each tracking device TA with multiple tracking devices TD (i.e., one or more tracking devices) continuously tracks at least one reference point RP and at least one marker point MP.

[0072] According to another embodiment of the tracking device TA, the tracking device TA may comprise a base that can be moved within the boundaries of the work site 13, whereby the position of the tracking device TA in the work site 13 can be easily changed. Such a tracking device TA may be, for example, a drone provided with two or more tracking devices TD. The control unit of the drone may be configured in such a way that the drone is not allowed to go beyond the boundaries of the work site 13. During the determination of the position and orientation of the machinery in the work site 13, the drone is preferably stationary. Alternatively, the drone may be stationary only when a threshold level of accuracy regarding the requested position and orientation of the machinery is not exceeded.

[0073] The base B of the tracking apparatus TA is adjustable, whereby the alignment of the tracking apparatus TA can be adjusted by adjusting the alignment of the base B. Hence, according to an embodiment of the positioning system PS, at least one tracking device TD may be mounted on an adjustable base. The adjustable base of the tracking device TD allows to conveniently adjust the orientation of the base B and thereby the orientation of the tracking device TD by taking into account, for example, the area in which the machine is to work and / or the position of the nearest reference point RP.

[0074] According to an embodiment of the positioning system, the tracking device TA comprises means for acquiring data about the orientation of each tracking device TD on the adjustable base B relative to the tracking device TA. When the orientation of the tracking device TD relative to the tracking device TA is known, the position and orientation of the machine can be accurately determined. The adjustment of the base B may be automatic or remote controlled, but may also be manually operated, whereby there may be graduations in the base B to indicate the orientation of the base B, for example selectable mounting points every 5 to 15 degrees in the direction of rotation, for example.

[0075] The positioning system PS further comprises at least one position determination unit PDU, i.e. one or more position determination units PDU. The position determination unit PDU comprises a receiving device for receiving data acquired by at least one tracking device TA. The data acquired by at least one tracking device TA comprises: identification data for identifying reference points RP and marker points MP and their positions relative to the tracking device TA (i.e. the position of the reference point RP identified relative to the detected position of the tracking device TA and the position of the marker point MP identified relative to the detected position of the tracking device TA), the identification data associating each specific reference point RP in the work site coordinate system WCS and each specific marker point MP in the machine coordinate system MCS with the corresponding position data of each reference point RP and each marker point MP relative to the tracking device TA.

[0076] The position relative to the tracking device position is, for example, a three-dimensional coordinate in the coordinate system of the tracking device. Alternatively, the position may be a three-dimensional coordinate in the work site coordinate system WCS and / or a three-dimensional coordinate in the machine coordinate system MCS.

[0077] The position determination unit PDU further comprises: a determination device for determining the position and orientation of the machine in the work site coordinate system WCS at least partially based on the received data (i.e. at least partially based on the data acquired by the tracking device TA). If there is also a machine coordinate system MCS fixed to the machine available, the position and orientation of the machine in the work site coordinate system WCS can be converted into the position and orientation of the work site in the machine coordinate system MCS in order to implement the work task to be performed by the machine.

[0078] At least one position determining unit PDU may be implemented by a combination of hardware and software. The implementation comprises an input / output unit for communicating with other devices connected to the position determining unit PDU, and a microprocessor or some other processing means capable of executing a computer program configured to process data received by the position determining unit PDU. The implementation may also comprise at least one storage unit for at least temporarily storing data received by the position determining unit PDU and / or data to be sent from the position determining unit PDU.

[0079] At least one position determining unit PDU may, for example, reside in any wired or wireless network, and / or in a computer accessible to the machine and / or at least one tracking device TA. When the position determining unit PDU resides in a computer accessible to any wired or wireless network, the physical location of the position determining unit PDU may be freely chosen, and thus the position determining unit PDU may be inside or outside the work site 13. When the position determining unit PDU resides in the machine, it may, for example, be implemented in the control unit 11 of the machine. When the position determining unit PDU resides in at least one tracking device TA, the tracking device TA is configured to include the necessary means to implement the operation of the position determining unit PDU.

[0080] In the case where at least one position determination unit PDU resides in a computer accessible to any wired or wireless network or in at least one tracking device TA, or in the case where at least one position determination unit PDU does not reside in a control unit of the machine, the machine includes at least one control unit, for example, a control unit 11, which obtains data about the position and orientation of the machine, wherein the control unit is configured to select at least one position determination unit PDU from which to receive the determined position and orientation of the machine in the work site 13.

[0081] According to an embodiment, at least one position determining unit PDU comprises receiving means for receiving data acquired by the machine. The data acquired by the machine may for example be about a position determined by at least one sensor SM( Figure 1 ), i.e., one or more sensors SM that may be installed in the machine to determine the position and / or direction and / or inclination and / or heading of the machine. The one or more sensors SM that may be installed in the machine will be discussed in more detail later.

[0082] When determining the position and orientation of the machine in the work site 13, at least one marker point MP is arranged on the machine and at least one reference point RP is arranged in the work site 13. The method further comprises: arranging at least one tracking device TA in the work site 13 to acquire data by tracking the position of the reference point RP and by tracking the position of the marker point MP relative to the tracking device TA. The data acquired by the tracking device TA is transmitted from the tracking device TA to the position determination unit PDU. The position determination unit PDU determines the position and orientation of the machine in the work site 13 at least partially based on the acquired data received from the tracking device TA. Figure 4 An embodiment of a method for determining the position and orientation of a machine in a work site 13 is schematically illustrated.

[0083] The installation of the tracking device TA at the work site 13 can be achieved simply by placing the tracking device TA in the work site 13, and then the tracking device TA itself can track or find at least one reference point RP and at least one marking point MP in the work site 13, and obtain data about the positions of the at least one reference point RP and at least one marking point MP in the work site 13. Therefore, unlike generally known tachometers or similar devices, there is no need to accurately adjust or position the tracking device TA in the work site 13.

[0084] Basically, the tracking device TA is configured to find all reference points RP and marker points MP within the operating range of the tracking device TA. If no reference point RP or any marker point MP is set in the work site 13, the tracking device TA cannot obtain data from at least one reference point RP and / or from at least one marker point MP, and will therefore not provide corresponding data about the position of at least one reference point RP and / or at least one marker point MP. In the case where the tracking device TA is arranged in the work site 13 and cannot identify any reference point RP and / or marker point MP, the tracking device can send a signal by any known means. In addition, the tracking device can send a signal by any known means how many reference points and / or marker points MP can be identified from its current position. Therefore, the person who arranges the tracking device TA in the work site can get feedback about where the person arranged the tracking device TA. The feedback can indicate the accuracy level that can be achieved for the current position of the tracking device TA.

[0085] The minimum requirements for the tracking device TA to start assisting in determining the position and orientation of the machine are to identify at least one reference point RP and track its position relative to the tracking device, and to identify at least one marker point MP and track its position relative to the tracking device, and to transmit data about the tracked reference point RP identification and the tracked marker point MP identification and their tracked positions relative to the tracking device to at least one position determination unit PDU. If the minimum requirements are not met, the tracking device TA may indicate it.

[0086] Furthermore, in addition to all the information transmitted by the tracking device TA to the position determination unit PDU, the position determination unit PDU needs to obtain the position information of the identified reference point RP in the work site coordinate system WCS and needs to obtain the position information of the identified marker point MP in the machine coordinate system MCS. The position information about the reference point RP in the work site coordinate system WCS and the position information about the marker point MP in the machine coordinate system MCS can be received by the position determination unit PDU in any known way.

[0087] In order to start determining the position and orientation of the machine, the position determining unit PDU may require additional information. The additional information required depends on how much information the position determining unit PDU receives from the tracking device TA.

[0088] Therefore, regarding the arrangement of the tracking device TA, it needs to be arranged so that it can identify at least one reference point RP and at least one marking point MP. The person who arranges the tracking device TA may be aware that the machinery has not arrived at the work site 13, so if the tracking device TA notifies it that it cannot find any marking point MP, the person can ignore the information. Alternatively, if a temporary barrier prevents the tracking device TA from identifying one or more reference points RP, and results in a situation where the tracking device TA cannot identify any reference point RP, the person can ignore the information and realize that tracking can only start after the temporary barrier is removed.

[0089] According to an embodiment of the positioning system PS, at least one tracking device TA comprises at least one sensor STA ( Figure 7 ), i.e. one or more sensors STA for determining the position and / or orientation and / or inclination and / or heading of the tracking device TA. Data comprising information about the position and / or orientation and / or inclination and / or heading of the tracking device TA are also considered to be data acquired by the tracking device TA, whereby the data acquired by the tracking device TA and transmitted to the position determination unit PDU may also include data comprising information about the position and / or orientation and / or inclination and / or heading of the tracking device TA.

[0090] According to an embodiment of the positioning system PS, the machine comprises at least one sensor SM ( Figure 1), i.e., one or more sensors SM for determining the position and / or direction and / or inclination and / or orientation of the machine. The sensors in the machine may also include sensors related to determining the orientation of the working tool of the machine. The effect of this embodiment is that in the absence of visual communication between the tracking device TA and at least one reference point RP in the work site 13 and / or at least one marking point MP in the machine (for example, due to an obstacle between the machine and at least one reference point RP and / or marking point MP), the machine can still continue (at least for a period of time) its operation due to the information obtained by at least one sensor SM in the machine. For example, before an obstacle occurs between the tracking device TA and at least one reference point RP and / or marking point MP, the machine can remain in its current state that has been determined by the position determination unit PDU and continue to determine or track the position of the working tool relative to the machine. How long the machine can continue its operation may depend on the accuracy required for the current task and the data obtained by the tracking device TA before the obstacle occurs between the machine and at least one reference point RP and / or at least one marking point MP. The time period may also depend on the accuracy of at least one sensor SM in the machine and how many sensors SM are present. Another effect related to this (the sensors in the machine may also include sensors related to determining the orientation of the working tool of the machine) is that if one or more marker points MP reside at a position that is not fixedly connected to the machine part (e.g. the origin of the machine coordinate system MCS positioning), the position determination unit PDU needs information about how this position is oriented relative to the machine coordinate system MCS or what the position of the marker point MP in the machine coordinate system MCS was at the exact tracking time. For example, such a position can be found from the boom 5 of the excavator 1, for example, Figure 1 The MP2s reside here.

[0091] One or more sensors in the mechanical and / or tracking device TA can be at least one of the following: a gyroscope, an accelerometer, an inclinometer, a magnetic compass, a satellite compass, an angle sensor, a position sensor, a pendulum, a level measuring device and any other sensor (for example, a camera sensor, a laser receiver / detector or a lidar), which is suitable for determining at least one of the position, location and orientation of at least one of an object and one or more objects attached to each other. Figure 8Some of these sensors are schematically shown. For example, when considering an excavator 1, the term object herein refers to the boom 5, its boom parts 5a, 5b and a working tool, such as a bucket 7. In the excavator 1, the sensors are preferably selected in such a way that the mutual orientation of the lower bracket 2a and the upper bracket 2b as well as the orientation of the boom 5 and / or its parts 5a, 5b and the working tool, such as a bucket 7, relative to the lower bracket 2a and the upper bracket 2b can be determined. For example, when considering an excavator 1, the term object herein refers to the mutual orientation of the tracking device TD in the tracking apparatus TA as well as the mutual orientation of the tracking device TD and its possible base B.

[0092] According to an embodiment, the tracking device TA further acquires data related to the stability of the tracking device TA. The stability of the tracking device describes the reliability of the data acquired by the tracking device.

[0093] According to an embodiment for acquiring data related to the stability of the tracking device TA, the tracking device TA comprises at least one gyroscope and / or at least one accelerometer for determining the stability of the tracking device TA. The at least one gyroscope and / or at least one accelerometer may be used, for example, to determine a swing or a shake of the tracking device TA, an amount of a swing or a shake of the tracking device TA characterizing the stability of the tracking device TA, and thus the stability of the tracking device TA describes the accuracy of the data acquired by the tracking device TA.

[0094] According to another embodiment for acquiring data related to the stability of the tracking device TA, it can be continuously observed from the data acquired by the tracking device TA about at least one reference point RP. According to such an embodiment, for example, the position determination unit PDU can be configured to determine the changes in the data acquired by the tracking device TA, and in the case of significant changes (i.e., higher than the accuracy required to perform the work), it can be assumed that at least one of the reference point RP and the tracking device TA is swinging or shaking, whereby the accuracy of the data acquired by the tracking device TA may be lower than expected. Whether it is significant depends on the accuracy currently required. For example, the accuracy of a specific work task can be set in a building information model (BIM model) set up for the work site 13. The position determination unit PDU can, for example, express the level of change in millimeters and / or degrees, and the operator can take it into account when the accuracy is reduced. Alternatively, the PDU can analyze the data acquired by the tracking device TA and indicate the cause of the change to the operator.

[0095] According to an embodiment, a tracking state of the tracking device TA is to be determined. The tracking state of the tracking device TA describes the current prevailing operating state of the tracking device TA. The tracking state of the tracking device TA may be determined based on data acquired by the tracking device TA. The operation of the tracking device TA may include at least the states: active, tracking, normal, pending and exited, one of which is prevailing. Fig. 9 Some possible tracking states of the tracking device TA are schematically shown.

[0096] When the tracking device TA is in an active state, the tracking device TA acquires data by tracking the positions of the reference points RP and the marker points MP and transmits the acquired data to the position determination unit PDU. However, the tracking device TA does not provide any indication about the accuracy, reliability or validity of the acquired data.

[0097] When the tracking device TA is in the tracking state, the tracking device TA acquires data by tracking the positions of the reference points RP and the marker points MP, and acquires data from the sensors STA installed on the tracking device TA, and transmits the acquired data to the position determination unit PDU. In addition, for example, based on the amount of acquired data and / or the accuracy of determination as disclosed above, the tracking device TA actively realizes that it can accurately (i.e., correctly enough) track any kind of point and / or location in the work site coordinate system WCS. Alternatively, based on the number and type of sensors STA (disclosed above) installed on the tracking device TA, the minimum number of reference points RP and / or marker points MP to be tracked in each case can be determined to be able to consider the tracking device TA to be in the tracking state. Therefore, if at least the minimum number of reference points RP and / or marker points MP are tracked, the tracking device is in the tracking state, otherwise it is in some other state.

[0098] When the tracking device TA is in the normal state, the tracking device TA acquires data by tracking the positions of the reference points RP and the marking points MP, and transmits the acquired data forward to the position determination unit PDU. In addition, the tracking device TA realizes that it is operating correctly, but for some reason, the correct operation of the tracking device TA cannot be verified. One of the reasons may be that when in the tracking state, the minimum number of tracked reference points RP is not met, but, for example, the sensors installed on the tracking device TA are able to verify that the stability of the tracking device TA has not changed to a lower level after the situation in which the tracked reference points RP fall below the minimum number occurs. The reason for this situation may be, for example, a person walking between one or more tracked reference points RP and the tracking device TA. If this situation lasts too long, the tracking state will be regarded as an activated state, or, for example, someone approaches the tracking device TA to turn it off.

[0099] When the tracking device TA is in the exit state, the tracking device TA stops operating. For example, it may have been turned off.

[0100] When the tracking device TA is in the pending state, the tracking device TA is initializing its operation and changing its state from the exit state to the tracking state or the active state.

[0101] According to an embodiment, the tracking device TA is configured to determine its tracking state, and the tracking device TA is further configured to transmit the tracking state and / or changes in the tracking state to the position determination unit PDU. Therefore, the tracking device TA may comprise necessary data processing means to identify its state and transmit state information thereto.

[0102] According to an embodiment, the availability of location-based features of a machine depends on the tracking state. According to this embodiment, depending on the tracking state of the tracking device TA, it is possible that no location data at all is available for the machine, or that the location data is insufficient for work tasks requiring high accuracy, whereby work tasks requiring the use of location data (with a required accuracy higher than the currently available accuracy) cannot be performed, but it is still possible to perform some other work tasks requiring an accuracy equal to or lower than the currently available accuracy. In addition, some work tasks may require high certainty at a minimum accuracy level, and therefore these work tasks may not be available if the tracking state is "no tracking". Alternatively, if the machine has a number of sensors SM assisting the position determination unit PDU, at least an activated and normal tracking state may also be sufficient, for example if said state has been tracked a short time ago.

[0103] According to an embodiment, the tracking state of the tracking device TA reaching a tracking accuracy level capable of accurately tracking any kind of point and / or location in the work site coordinate system WCS can be determined by at least one of the following: a) acquiring data by tracking at least three reference points RP related to the tracking device TA; b) acquiring data by tracking one to two reference points RP related to the tracking device TA in addition to acquiring data related to the inclination of the tracking device TA, and c) acquiring data by determining the positions of at least four satellites relative to the tracking device TA.

[0104] According to an embodiment, therefore, a tracking state in which the tracking device TA reaches a level of tracking accuracy capable of accurately tracking any kind of point and / or location in the work site coordinate system WCS can be determined by acquiring data by tracking at least three reference points RP relative to the tracking device TA. These reference points may not be located on a line observed in three dimensions. The farther the reference point RP is from a straight line in three-dimensional space, the better the accuracy achieved. If the tracking device TA does not include any sensor STA for determining the inclination of the tracking device TA, at least three reference points RP that are not located on a straight line in three-dimensional space are needed so that the necessary data are available to achieve a level of being able to track the machine in the work site coordinate system. As discussed above, this state may be referred to as tracking.

[0105] In addition to or in lieu of acquiring data by tracking at least three reference points RP relative to the tracking device TA, a tracking state of the tracking device TA that achieves a tracking accuracy level capable of accurately tracking any kind of point and / or location in the work site coordinate system WCS may be determined by acquiring data by tracking two reference points RP relative to the tracking device TA in addition to acquiring data related to at least one of the inclination of the tracking device TA and the direction to the north from the tracking device TA. Therefore, in this embodiment, the inclination information of the tracking device TA and / or the direction to the north from the tracking device TA is acquired by, for example, some of the methods described above, whereby tracking only two reference points RP is sufficient to determine that the tracking device TA achieves a tracking state capable of tracking the machinery in the work site coordinate system WCS.

[0106] It should be noted that in the case where the inclination of the tracking device TA is used to track two reference points RP, the reference points may not reside above or parallel to the other with respect to the Earth's gravitational field - if they do, the direction to North remains unresolved. Therefore, in this case, the greater the distance (measured in angles) between the reference points, or the more perpendicular they are, the better the accuracy obtained.

[0107] Furthermore, it should be noted that in case the direction north from the tracking device TA is used to track the two reference points RP, the reference points may not reside at the same altitude from the Earth, meaning at the same level relative to the Earth's gravitational field - if they do, then the inclination of the tracking device TA remains unresolved. Therefore, in this case, the further (measured in angles) the reference points RP are from being at the same altitude from the Earth (i.e. at the same level relative to the Earth's gravitational field), or the more parallel they are to it, the better the accuracy obtained.

[0108] In addition to or in lieu of data acquired by tracking at least three reference points RP relative to the tracking device TA, and / or in addition to acquiring data related to at least one of the inclination of the tracking device TA and the direction to the north from the tracking device TA, by tracking two reference points RP relative to the tracking device TA, a tracking state of the tracking device TA reaching a tracking accuracy level capable of accurately tracking any kind of point and / or location in the work site coordinate system WCS may be determined by: acquiring data by tracking one reference point RP relative to the tracking device TA in addition to acquiring data related to the inclination of the tracking device and data related to the direction to the north from the tracking device. Therefore, in the present embodiment, the inclination information of the tracking device TA and the direction to the north from the tracking device TA are acquired by, for example, some of the methods described above, whereby tracking only one reference point RP is sufficient to determine that the tracking device TA reaches a tracking state capable of tracking the machinery in the work site coordinate system WCS.

[0109] In addition to or in lieu of data acquired by tracking at least three reference points RP relative to the tracking device TA, and / or data acquired by tracking two reference points RP relative to the tracking device TA in addition to acquiring data related to at least one of the tilt of the tracking device TA and the direction to the north from the tracking device TA, and / or data acquired by tracking one reference point RP relative to the tracking device TA in addition to acquiring data related to the tilt of the tracking device and acquiring data related to the direction to the north from the tracking device, a tracking state of the tracking device TA reaching a tracking accuracy level capable of accurately tracking any kind of point and / or location in the work site coordinate system WCS can be determined by acquiring data by determining the positions of at least four satellites relative to the tracking device TA. In the present embodiment, at least two antennas 12 in the tracking device TA determine the positions of at least four satellites relative to the tracking device TA. Figure 7 A tracking device TA with two antennas 12 is schematically shown on the right. When there are less than four satellites, the acquired data cannot provide accurate information unless the tracking device TA includes more than two antennas 12. In this embodiment, the tracking device TA can be equipped with at least two antennas 12, instead of equipping all machines in the work site with at least two antennas 12. In addition, with respect to this embodiment, according to which the tracking device TA is equipped with at least two antennas 12, at least one of the at least two antennas 12 can reside in the work site 13. In this embodiment, it is necessary to optically locate the antenna 12 residing in the work site 13 relative to the tracking device TA, for example, by using at least one of the tracking devices TD of the tracking device TA.

[0110] According to the above-described embodiment regarding the tracking device TA reaching a level (i.e., tracking state) at which the machine can be tracked in the work site coordinate system WCS, after reaching the level, if the tracking device TA can determine that it has maintained its stability, the level can be maintained despite not meeting the minimum requirements at every moment. Similarly, according to the above-described embodiment, it is possible to detect whether the tracking state has changed or whether the tracking state is in a pending state or an exit state.

[0111] According to an embodiment, data is acquired by tracking the position of one or more reference points RP relative to the tracking device TA, accompanied by semi-automatic and / or automatic identification of the initial position of the one or more reference points RP.

[0112] When the initial position of one or more reference points RP is semi-automatically identified, the operator 10 finds at least one reference point RP at the work site 13 and directs the tracking device TA to focus on the at least one reference point RP. Alternatively, the operator 10 may point the tracking device TA to an area including at least one reference point RP, and the tracking device TA itself identifies at least one reference point RP at the work site 13 and focuses on at least one reference point RP. For example, the operator 10 may select at least one reference point RP as the specific at least one identified reference point RP in a menu or database of reference points RP, or the positioning system PS itself identifies at least one reference point RP in a menu or database of reference points RP. For example, the menu or database of reference points RP may be retrieved from a cloud service into the control unit 11 of the excavator 1, whereby they may be retrieved from the excavator 1 via at least one position determination unit PDU.

[0113] When the initial position of one or more reference points RP is automatically identified, the tracking device TA itself identifies the necessary number of reference points RP at the work site 13 and assigns to them information from a database comprising work site information.

[0114] Semi-automatic and / or automatic identification of the initial position of one or more reference points RP may be controlled by the control unit 11 of the excavator 1 and / or by the tracking device TA and / or by the position determination unit PDU. The identification of the initial position of one or more reference points RP is performed so as to individualize the reference points RP to the position determination unit PDU and retrieve the position information about each identified reference point RP so that the position determination unit PDU can operate correctly. The position information about each reference point RP in the work site 13 may be input using wireless or wired I / O devices and / or may be retrieved from any known location (e.g., a work site computer, a cloud service, and / or any computer or storage medium accessible by any wired or wireless network).

[0115] According to an embodiment, the tracking device TA further determines the position and orientation of the tracking device TA itself based on the acquired data, and the acquired data transmitted from the tracking device TA to at least one position determination unit PDU includes at least: position and orientation data of the tracking device TA, tracking data about at least three markers MP, and data from which the tracking state of the tracking device TA can be determined. The tracking device TA may be able to determine its position and orientation, for example, using the sensors disclosed above. Thereafter, the position determination unit PDU determines the position and orientation of the machine based on the acquired data transmitted from the tracking device to the position determination unit PDU. This embodiment is applicable to such a machine: it does not include any sensor, or only includes a few sensors related to determining the position and orientation of the machine therein, but includes a position determination unit PDU. Based on the features disclosed herein, the machine, especially the position determination unit PDU therein, receives sufficient data to be able to determine the position and orientation of the machine and the accuracy of the determined position and orientation. According to this embodiment, the tracking device TA predetermines its own position so that it does not need to be instantly aware of the data based on the position and orientation determined by it. In addition, according to this embodiment, the tracking device TA can be used as a temporary reference point for other tracking devices TA. In this case, the temporary reference point should indicate by some well-known means when it can be used as a temporary reference point. Therefore, the tracking device may include a corresponding temporary reference marker, which includes at least one temporary reference point, and thus, the position information should be available, and similarly, the position information about each reference point RP disclosed above is also available.

[0116] According to an embodiment, the position determination unit PDU further determines the accuracy level of the determined position and orientation of the machine, and based on the determined accuracy level, the machine provides at least one of the following options: a) enabling the selected operating mode at the current accuracy level; b) instructing the operator if the current accuracy level is lower than or falls below a threshold level for the selected operating mode of the machine; and c) disabling the operating mode that requires a more accurate position and orientation of the machine. According to this embodiment, the operating mode of the machine (i.e., the work task of the machine) can be classified based on the accuracy of the position and orientation of the machine required for the machine to perform a specific work task. If the current position or orientation accuracy of the machine is not sufficient to perform a specific work task, the machine is prevented from performing the specific work task until the position and orientation accuracy is high enough, or at least the operator can be informed that the specific work task may not be recorded and / or the specific work task is considered to be completed because the position and orientation accuracy is not higher than the threshold level required for the work task. It is considered that the higher the position accuracy, the more reference point RP and mark point MP data are acquired, the newer the data, and the more general the acquired data. Position accuracy also depends on how close the reference points RP and marker points MP whose data are acquired are to each other, their positions relative to each other, and what other sensor information (such as the inclination and / or direction to north disclosed above) is available.

[0117] The acquired reference point RP and marker point MP data are generic if there is position and orientation data for both reference point RP and marker point MP that are both substantially close to the tracking device TA and substantially far from the tracking device TA. In addition to the information contained in the reference point RP and marker point MP data, the positioning accuracy may also depend on possible information about the stability and inclination of the machine received from the machine and its sensors.

[0118] According to an embodiment of the positioning system PS, at least one tracking device TA comprises at least one tachometer, and at least one marker point MP is a prism or a tag detectable by the tachometer. Furthermore, according to this embodiment, the machine comprises at least one gyroscope and / or at least one accelerometer located in a known position relative to at least one marker point MP, and at least one position determination unit PDU further comprises receiving means for receiving data related to the position of at least one gyroscope and / or at least one accelerometer relative to at least one marker point MP, and for receiving data from at least one gyroscope and / or at least one accelerometer. The gyroscope in the machine provides information about changes in the orientation of the machine and, in some cases, the direction to the north, the gyroscope being able to learn the orientation of, for example, the upper bracket 2b of the excavator 1 by rotating the upper bracket 2b several times relative to the lower bracket 2a. The accelerometer in the machine provides information about the direction of the earth's gravitational field and, therefore, about the inclination of the machine. Therefore, in this embodiment, there are also available data from the gyroscope and / or the accelerometer, which can be used to define changes in the position and orientation of the machine between operating cycles of the tachometer.

[0119] According to an embodiment of the method for determining the position and orientation of a machine in a work site 13, the work site 13 is provided with at least one reference point RP, and the method comprises: providing a tracking device TA on the machine, tracking the machine with the tracking device TA by determining the position of at least one reference point RP in the work site 13 relative to the tracking device TA, transmitting data from the tracking device TA to a position determination unit PDU associated with the tracking, and determining the position and orientation of the machine in the work site 13 by the position determination unit PDU at least partially based on the data received from the tracking device TA. According to this embodiment, the machine itself (e.g., the excavator 1) is provided with a tracking device TA, which is arranged to track the machine by determining the position of at least one reference point RP in the work site 13 relative to the tracking device TA. Fig.10 Schematically a top view of a work site 13 is shown, wherein a machine is provided with a tracking apparatus TA, whereby there is a visual connection TD_RP between a tracking device TD of the tracking apparatus TA and a reference point RP at a reference mark RM. Fig.11 Schematically illustrated is a method according to the present embodiment for determining the position and orientation of a machine in a work site 13. In the present embodiment, the marking points MP intended to be set to the machine can be omitted or one or more marking points MP known in the machine coordinate system MCS can be used to verify whether the tracking device TA reattached to the machine resides in the same position and orientation as when it was previously attached, and thus, making it possible to detach the tracking device TA after a day's work.

[0120] According to an embodiment, the tracking device TD may be a laser radar, which may be a mechanical laser radar or a solid-state laser radar. Due to the narrower field of view of a solid-state laser radar compared to a mechanical laser radar, the number of solid-state laser radars required for a particular application may be higher than the number of mechanical laser radars. However, when compared to the price of a mechanical laser radar, the higher number of solid-state laser radars required may be compensated by their ultra-low price.

[0121] If the tracking device TD is a laser radar, the corresponding reference point RP is selected so that the laser radar can detect the reference point RP. For example, the reference point RP may include a plurality of balls, that is, one or more balls. If a single reference marker RM includes a single ball as the reference point RP, different reference markers RM at the work site 13 may include balls of different sizes relative to other reference markers RM, so that each reference marker RM and the corresponding reference point RP are unique relative to the other reference markers RM and the corresponding reference point RP therein. For example, the diameter of the ball forming the corresponding reference point RP can be selected as 5 cm, 10 cm, 15 cm, ..., or 2 cm, 4 cm, 6 cm, 8 cm, ..., or 3 cm, 6 cm, 9 cm, 12 cm, ..., etc. The size of the ball and the difference in diameter between them can be selected according to the actual distance between the tracking device TD and the reference point RP at the work site 13.

[0122] According to an embodiment, if there are several reference marks and corresponding reference points RP at the work site 13, each reference point RP may include many balls of the same size but different arrangements, or many balls of different sizes but the same or different arrangements, so that each reference point RP is unique, so that the reference point RP can be distinguished from other reference points RP.

[0123] According to an embodiment, if there are several reference markers RM and corresponding reference points RP at the work site 13, each reference point RP may include a number of balls of the same size and, in addition, a number of balls (i.e., one or more balls) having different sizes to distinguish each reference point RP from the other reference points RP. Instead of one or more balls of different sizes, the reference points RP may be distinguished from each other by a code applied to the reference point. For example, the code may be mechanically readable by an appropriate sensor, or the code may be manually stored in an appropriate menu or database, whereby the position of a specific reference point RP at the work site 13 may be determined based on the code. The code of the reference point RP and the position of the corresponding reference point RP may be indicated to the positioning system PS, so that the positioning system PS can distinguish the reference points RP from each other with a substantially moderate level of accuracy even after the first successful positioning, thereby preventing the need to reposition them after each change in their alignment.

[0124] According to an embodiment of the method, wherein the machine is provided with a tracking device TA, the method further comprises initializing a tracking state of the tracking device TA by determining a position and an orientation of the tracking device TA in a machine coordinate system MCS. According to this embodiment, the tracking device TA may be introduced by arranging the tracking device TA on the machine and allowing the tracking device TA to check or verify its position relative to the machine.

[0125] According to an embodiment of the method, wherein the machine is provided with a tracking device TA, the method further comprises indicating, by the position determination unit PDU, a current level of accuracy relating to the position and orientation of the machine as implemented by data received from the tracking device TA. The level of accuracy relating to the position and orientation of the machine as implemented by data received from the tracking device TA, if the level of accuracy is low, may result in that work tasks requiring a high level of accuracy cannot be performed or completed, as long as the level of accuracy is not high enough.

[0126] According to an embodiment of the method, wherein the machine is provided with a tracking device TA, the method further comprises: determining a level of accuracy with respect to the position and orientation of the machine to be achieved, i.e., being above a predetermined minimum threshold level; detecting, by a position determination unit, the need for a higher level of accuracy with respect to the position and orientation of the machine; and acquiring, by the position determination unit, additional tracking data from the tracking device. According to this embodiment, the level of accuracy with respect to the position and orientation of the machine to be achieved may be determined as a minimum threshold level based on a specific work task to be performed by the machine, or based on information provided by an operator, for example, by a control unit of the machine, a position determination unit PDU or a building information modeling (BIM) model residing, for example, in a cloud service or a work site computer. Thereafter, the position determination unit PDU may detect the need for a higher level of accuracy with respect to the position and orientation of the machine and acquire additional tracking data from the tracking device TA. The additional tracking data may be acquired semi-automatically and / or automatically from the tracking device TA.

[0127] Semi-automatically acquiring additional tracking data from the tracking device TA comprises: arranging the position determination unit PDU to provide at least one of the following: indicating to the operator 10 that the tracking device TA is required to detect at least one reference point RP, indicating to the operator that the tracking device TA is required to detect another reference point RP, and indicating to the operator that the tracking device TA is required to detect at least one further reference point RP, whereby the operator can operate the machine according to said indication. Thus, acquiring additional tracking data from the tracking device TA comprises retrieving information from the position and orientation of at least one reference point RP in case the tracking device TA is unable to detect any reference point RP, or retrieving information from the position and orientation of at least one additional reference point RP in case the reference point RP already tracked by the tracking device TA does not provide sufficient information to determine the position and orientation of the machine in the work site 13 with sufficient accuracy.

[0128] According to an embodiment of the method for semi-automatically acquiring additional tracking data from a tracking device TA, the setting of the tracking device TA on the machine comprises setting the tracking device TA on the machine on an adjustable base B, whereby the operator 10 can operate the adjustable base B according to instructions received from the position determination unit PDU and initialize the tracking state of the tracking device TA by determining the position and orientation of the tracking device TA in the machine coordinate system MCS after each operation of the adjustable base B. According to this embodiment, the operator 10 can adjust the tracking device TA by operating the adjustable base B to find or locate at least one reference point RP intended to be tracked by the tracking device TA.

[0129] According to an embodiment of the method for automatically acquiring additional tracking data from a tracking device TA, the setting of the tracking device TA on a machine comprises: setting the tracking device TA on the machine on an adjustable base B, and automatically acquiring additional tracking data from the tracking device TA by controlling the adjustable base B by at least one of the tracking device TA and the position determination unit PDU. According to this embodiment, at least one of the tracking device TA and the position determination unit PDU is configured to adjust the adjustable base B so that the tracking device TA finds or locates at least one reference point RP intended to be tracked by the tracking device TA.

[0130] According to an embodiment of the method, wherein the machine is provided with a tracking device TA, the method further comprises: arranging one or more additional tracking devices TA to the work site 13 and equipping the machine with at least one marker point MP known in the machine coordinate system MCS; acquiring data by the one or more additional tracking devices TA by tracking the reference point RP and the marker point MP position relative to the corresponding one or more additional tracking devices TA; transmitting the data acquired from the one or more additional tracking devices TA to at least one position determination unit PDU; and determining the position and orientation of the machine in the work site coordinate system WCS by the at least one position determination unit PDU at least partially based on the data received from the tracking device and at least one of the one or more additional tracking devices. In other words, in this embodiment, one or more additional tracking devices TA are arranged at the work site 13, and at least one marker point MP is arranged on the machine, the marker point MP being intended to be tracked by the at least one additional tracking device TA. Fig.12A top view of a work site 13 is schematically shown, wherein a machine is provided with a first tracking device TA1, and a second tracking device TA2 is provided at the work site 13, whereby a visual connection TD1_RP exists between a first tracking device TD1 in the first tracking device TA1 provided on the machine and a reference point RP at a reference mark RM in the work site 13, and a visual connection TD2_MP exists between a second tracking device TD2 in a second tracking device TA2 provided in the work site 13 and a mark MP at a mark MA provided in the machine, and a visual connection TD2_RP exists between a second tracking device TD2 in the second tracking device TA2 provided in the work site 13 and a reference point RP at a reference mark RM in the work site 13.

[0131] According to an embodiment of the method comprising one or more additional tracking devices TA, the determination of the position and orientation of the tracking device TA in the machine coordinate system MCS may be determined using one of the additional tracking devices TA. According to this embodiment, the determination of the position and orientation of the tracking device TA arranged in the machine may be determined in the machine coordinate system MCS using one of the additional tracking devices TA.

[0132] Some applications for determining, in particular, the position and orientation of a machine in a work site 13 have been described above. In order to gain valuable knowledge about the condition or state of the work site 13, whether near the work machine or far from the work machine, such as at an area of ​​a portion of the work site 13 or at an entire area of ​​the work site, additional measures may be combined with determining the position and orientation of the machine in the work site 13 to determine situational awareness in the work site 13. Situational awareness in the work site 13 is: knowledge of the position and orientation of the work machine in the work site 13, knowledge of the position and orientation of any other work machine, knowledge of materials, tools, personnel, animals, or any objects residing and / or moving in the work site, and knowledge of the condition or state of the work site 13. Without accurate knowledge of the position and orientation of the work machine in the work site 13, equipment residing in the machine may not be able to determine the current condition or state of the work site or a portion thereof, and thus situational awareness in the work site or a portion thereof. On the other hand, if situational awareness about the condition or state of the work site has been accurately determined, the machine can use this information to help accurately determine its position and orientation in the work site. In the following embodiments, solutions for determining situational awareness in a work site 13 have been presented, wherein the determination of the position and orientation of a machine in the work site 13 can be used to control the machine by using information describing knowledge of the current condition or state of the work site 13. Furthermore, determining the position and orientation of a machine can be an important part of situational awareness for some other machine, object, or person, as data regarding the determined position and orientation of the machine can be made available to others as situational awareness.

[0133] According to an embodiment of the method for determining situational awareness in a work site 13, at least one environment modeling device EM is arranged at least one of the following: on the machine or outside the machine, that is, arranged on the machine and / or outside the machine. In addition, at least one tracking device TA is arranged at least one of the following: on the machine or outside the machine, that is, arranged on the machine and / or outside the machine. Thereafter, data is acquired by the at least one tracking device TA and by the at least one environment modeling device EM. Data related to the at least one tracking device TA and data related to the at least one environment modeling device EM are received by at least one position determination unit PDU. At least partly based on the received data, the position and orientation of the machine in the work site 13 are determined by the at least one position determination unit PDU. Similar embodiments are described in detail in the accompanying drawings. Fig.13 , Fig.14 and Fig.15 Schematically shown, where Fig.13 The top view of the fourth work site 13 is schematically shown. Fig.14Some components of the positioning system PS that are also used to determine situational awareness in the work site 13 are schematically shown, and Fig.15 Schematically illustrated are some steps of a method for determining situational awareness in a work site 13. It should be noted that data related to at least one tracking device and / or at least one environmental modeling device may be used by one or more position determining units PDU of one or more machines, whereby at least one tracking device and / or at least one environmental modeling device arranged in one machine may be exploited or utilized by at least one other machine.

[0134] Fig.13 A top view of a fourth work site 13 with a machine is schematically shown, the machine being an excavator 1 and provided with a first tracking device TA1, and the work site 13 is provided with a second tracking device TA2, so that: there is a visual connection TD1_RP between a first tracking device TD1 in the first tracking device TA1 provided on the machine and a reference point RP at a reference mark RM provided in the work site 13, and there is a visual connection TD2_MP between a second tracking device TD2 in the second tracking device TA2 provided in the work site 13 and a marking point MP at a mark MA provided in the machine, and there is a visual connection TD2_RP between a second tracking device TD2 in the second tracking device TA2 provided in the work site 13 and a reference point RP at a reference mark RM provided in the work site 13. The first tracking device TA1 represents a tracking device provided on the machine, and the second tracking device TA2 represents a tracking device provided outside the machine. Fig.13 In the embodiment of FIG. 1 , the second tracking device TA2 is disposed in the work site 13 , but the second tracking device TA2 may also be disposed on another machine and thus may also be movable in the work site 13 . Fig.13 In an embodiment, the tracking device TA1, TA2 is arranged to acquire data for determining the position and orientation of the machine in the work site 13. According to an embodiment, furthermore, only one tracking device TA1, TA2 is available.

[0135] Fig.13The embodiment also comprises a number of environment modeling devices EM. To be more specific, there is a first environment modeling device EM1 and a second environment modeling device EM2. The first environment modeling device EM1 is arranged on the machine. The second environment modeling device EM2 is outside the machine and is arranged in the work site 13, but the second environment modeling device EM2 can also be arranged on another machine and thus may also be able to move in the work site 13. According to an embodiment, moreover, only one environment modeling device EM1, EM2 is available. The environment modeling devices EM1, EM2 are arranged to acquire data related to the work site 13 or a part thereof. The data can be used at least in part to determine the position and orientation of the machine or any other machine in the work site 13.

[0136] exist Fig.13 In the example of FIG. 1 , the first environment modeling device EM1 is arranged to acquire data related to the first object OB1 in the work site 13 through a first data acquisition connection EM1_OB1 between the first environment modeling device EM1 and the first object OB1. The first environment modeling device EM1 is also arranged to acquire data related to the second object OB2 in the work site 13 through a second data acquisition connection EM1_OB2 between the first environment modeling device EM1 and the second object OB2. The second environment modeling device EM2 is also arranged to acquire data related to the second object OB2 in the work site 13 through a third data acquisition connection EM2_OB2 between the second environment modeling device EM2 and the second object OB2. Fig.13 The corresponding data acquisition connections are schematically shown in FIG. The data acquired by the at least one environment modeling device EM1, EM2 is used to provide a model of the work site 13 depicting the current or present state of the work site 13 or a part thereof. The model may be a geo-referenced spatial data model, or also referred to as a model later, in which data describing the current or present state of the work site 13 are available.

[0137] According to an embodiment, in the case where at least one reference point and / or at least one marker point can be tracked and / or the surrounding area can be detected by the same device as spatial data of the work site or data related to the work site, at least one tracking device TA and at least one environment modeling device EM can be the same device, such as a stereo camera, depending on the actual device providing the data acquisition. Tracking the reference points and / or the marker points can be performed simultaneously and / or at different times with the following: detecting the surrounding area as spatial data of the work or spatial data related to the work, and saving the saved part of the detected spatial data to the model.

[0138] The environment modeling device EM1, EM2 may include, for example, a camera, a stereo camera, a lidar, a radar or a speedometer as a modeling or tracking device. Thus, the data acquisition connection between the environment modeling device and the corresponding object provided by the environment modeling device may be visual or non-visual. For example, in the case where the determination of the position and orientation of the machine by using data acquired by at least one tracking device TA and / or at least one environment modeling device EM is prevented or considered unreliable or prevented from providing useful additional information (for example, for determining the expected and / or unexpected movement of the machine), in combination with the environment modeling device, there is also an inertial measurement unit, which includes, for example, an accelerometer, a gyroscope or a magnetometer or at least one of some other sensors disclosed above, which can be applied to determine the angular rate and / or orientation of the environment modeling device and / or the forces affecting it, so as to maintain or improve the determination of the position and orientation of the machine. In addition, the environment modeling device EM either includes or is connected to a device allowing the reception and / or transmission of information, such as an input-output unit.

[0139] The first object OB1 and the second object OB2 are objects in the work site 13 that can be influenced or taken into account when controlling the operation of the machine. Thus, they can be objects that have only some influence on the control of the machine, or objects that are actively influenced by the operation performed by the machine. The objects OB1, OB2 can be located in the work site 13 at a part of the work site 13, where active work to be performed by the machine is about to be performed. Alternatively, at least one object OB1, OB2 can be located in the work site 13 outside a part of the work site 13, where active work to be performed by the machine is about to be performed, but the part may still have some indirect influence on the work to be performed by the machine, and / or may be indirectly influenced by the work to be performed by the machine at the part of the work site 13, where active work is about to be performed by the machine. Thus, the objects can be objects of fixed permanent objects, such as rocks, rock masses or buildings, which are intended to remain in place in the work site 13, or to be removed from the work site 13, or objects constructed therein, or just the surface contour of the ground. Alternatively, the objects may be objects that are only temporarily arranged at some location in the work site 13, but will be removed to another location later, such as materials to be used in the relevant construction work. The objects may also be objects that form part of the machine positioning system PS, such as reference markers RM or markers MA tracked by or at least one tracking device and / or at least one environmental modeling device. There may be any number of such objects to be monitored or tracked in the work site 13. In order to determine situational awareness, the current or present state of these objects and their progress may be modeled by equipment or devices providing environmental modeling.

[0140] Fig.14 Schematically shows the Fig.13 and Fig.15 The positioning system PS used in this article is basically similar to Figure 3 As shown, but additionally includes components related to the determination of situational awareness, such as an environment modeling device EM and an environment modeling unit EMU providing a central unit for many environment modeling devices EM.

[0141] The at least one tracking device TA is arranged to acquire data relating in particular to the determination of the position and orientation of the machine in the work site 13. The data relating to the at least one tracking device TA are received in the position determination unit PDU by means of receiving means for receiving data relating to the at least one tracking device. The data relating to the at least one tracking device TA may comprise, for example, data acquired by the tracking device TA and data relating to the inclination and / or orientation and / or position information of the respective at least one tracking device TA itself, which data the machine may use to determine its own position and / or orientation. The data relating to the at least one tracking device will be considered in more detail later in this description.

[0142] The data related to at least one environmental modeling device EM, which will be considered in more detail later in this specification, may include: data acquired by the at least one environmental modeling device EM, and / or the inclination and / or direction and / or position information of the corresponding at least one environmental modeling device EM itself, and / or available early data about the work site. The available early data about the work site may be, for example, a building information model (BIM model) or spatial data of the work site or spatial data related to the work site, or a combination of previous data. The spatial data of the work site or spatial data related to the work site or geographically referenced spatial data may be, for example, data describing the current or present state of the work site or a part thereof, or more generally, data that implicitly or explicitly references a position relative to at least one of the work site 13 or the earth. Thus, such data may be, for example, data about a truck that transports gravel to the work site by sharing its position or time of arrival at the work site to the model, and thus sharing the data may influence how the excavator operator plans to use his / her working time in the next few minutes or hours. Furthermore, the spatial data that is shared and considered relevant to the work site may also be additional information about the crushed stone, such as its color, total weight of the load, moisture, temperature, etc.

[0143] The current or present state of the work site may include at least information about the working period, operation or phase of each area of ​​the work site, wherein the work site may have been divided into necessary small areas, for example, a pile of materials may constitute an area with known working periods and additional information about the pile of materials. The current or present state of the work site may further include information about objects and / or working machinery in the work site or information associated with the work site. For example, a pile of materials ordered and arriving at the work site at a certain time requires an area to be placed at the work site. The current or present state of the work site may further include the personnel working in the work site, and each person may be associated with location data.

[0144] The geo-referenced spatial data may be formed into, for example, a database, a data structure and / or a model. Alternatively, the geo-referenced spatial data may be formed and / or organized and / or structured into a database having a specific data structure and / or model. The model refers to a geo-referenced spatial data model, also referred to as a model below. For example, the model may be similar to a BIM model. The database, data structure and / or model are designed to interact with an application and / or a user and are accessed by an authorized user and / or an authorized operator and / or an authorized system and / or an authorized application. For example, data may be obtained from many tracking devices, environmental modeling devices, and measuring devices commonly used in a work site, and the data may include, for example, a reference point RP of a work site, a marking point of a machine running in the work site or associated with the work site, or any other traceable marker, the marker being attached to at least one of the following: a machine, an obstacle, or equipment of a person working on the work site. The operator may obtain identification data about a reference point RP, a marking point MP, and any other traceable marker via a database, a data structure, or a model. Data relating to the at least one environment modeling device EM are received in the position determining unit PDU by means of receiving means for receiving data relating to the at least one environment modeling device EM. The position determining unit PDU is configured to determine the position and orientation of the machine in the work site 13 at least partly based on the received data, i.e. data relating to the at least one tracking device TA and the at least one environment modeling device EM. In addition to the received data, i.e. data relating to the tracking device and the environment modeling device, the position determining unit PDU may also receive data from the machine by means of receiving means for receiving data acquired by the machine, which data may be taken into account when the position determining unit PDU determines the position and orientation of the machine in the work site 13.

[0145] The above solution provides for the determination of the position and orientation of the machine in the work site. This means that after the position and orientation are determined at the work site, the machine can provide its position and orientation information as geo-referenced spatial data to other operators in the work site 13, such as other tracking devices and / or environmental modeling devices, to be used as situational awareness information when acquiring data. On the other hand, after the position and orientation of the machine in the work site is determined, the machine itself is aware of its position and orientation in the geo-referenced spatial data, so it can observe its surroundings in the geo-referenced spatial data model and perform work tasks according to the BIM model, and update the geo-referenced spatial data model about the work tasks performed, and observe the position and orientation of the machine that provides its position and orientation information to the geo-referenced spatial data model or model. As initial data for starting the procedure of determining the position and orientation of the machine and then providing itself and other operators and the model with situational awareness in the work site 13 according to the system installation of the machine (e.g., number of sensors, etc.), the position and orientation of at least one reference point RP detected by the tracking device TA is required, or the environmental modeling device EM can detect geo-referenced spatial data that is marked or confirmed to be sufficiently accurate to be able to determine the position and orientation of the machine in the work site 13. Thereafter, determining the position and orientation in the BIM model of the work site 13 may be performed and providing situational awareness of the machine to the model for use by other operators, as well as for updating the model and / or the BIM model regarding the work tasks being performed. Although the machine is typically located in a portion of the work site where it is about to perform active work, sometimes at least a portion of the machine may extend beyond the portion in the work site, for example, to reach materials used in a construction project. In response to the utilization of at least one tracking device TA and at least one environmental modeling device EM, determining the position and orientation of the machine in the work site may also be performed in such cases. It should be noted that the position and orientation of the machine can generally be determined with sufficient accuracy, although not both the tracking device TA and the environmental modeling device EM are capable of acquiring valuable data regarding determining the position and orientation of the machine. Typically, there is a sufficient combination of data from sensors of the machine, data from sensors of at least one tracking device TA, data from sensors of at least one environmental modeling device EM, data acquired by at least one tracking device TA, and data acquired by at least one environmental modeling device EM to result in a sufficiently accurate determination of the position and orientation of the machine. For example, many different combinations enable a passerby walking or standing in front of at least one tracking device TA or at least one environmental modeling device EM to not be unable to determine the position and orientation of the machinery in the work site and thereby interrupt a work task requiring the accurate position and orientation of the machinery, or be unable to determine situational awareness regarding the machinery in the work site.Furthermore, by utilizing at least one environmental modeling device EM, it is also possible to obtain data on how it may be affected in response to the work performed by the machine in the part of the work site where the active work is to be performed and the objects surrounding it, thereby also making it possible to detect various negative effects, such as down-dip faults, that may occur at the work site 13. The solution is applicable to all operating situations of the machine, i.e. a machine that is substantially stationary when performing work, a machine that works at the same location but does not substantially move from one location to another, and a machine that actively or substantially actively moves from one location to another.

[0146] According to an embodiment, the driving direction or an alternative driving direction of the machine in the work site 13 is determined by at least one position determination unit PDU, at least partly based on the received data. An effect of this embodiment is that the exact driving direction of the moving machine can be determined based on the data acquired by at least one environment modeling device EM and by at least one tracking device TA. If the machine is an excavator, the excavator has two main driving directions, both directions additionally with or without simultaneous turning, all of which options can be detected at least partly based on the received data. Another effect of this embodiment is that the situational awareness data created by the machine may include information in the model about how fast the machine can move in which direction. Thus, for example, an automated vehicle working near the machine will choose a route passing the machine, which route may not be interrupted by the machine. In addition, if a person is in a sector of the machine, which is one of the sectors in which the machine can move quickly, he / she may be informed or warned, for example because he / she is not paying attention because he / she is against the machine.

[0147] According to an embodiment, at least one of a level of accuracy or validity of the position and orientation determined by the machine in the work site 13, ie, a level of accuracy and / or validity, is further determined.

[0148] The achievable level of accuracy of the position and orientation of the machine determined in the work site may be influenced by the number of environmental modeling devices used, by the number of tracking devices used, and by the number of reference points and marking points used, and by the area used to detect the surrounding environment, and by the distance between the tracked position and the tracking device, and also by other factors as described above in this specification, such as by the stability of the tracking device TA. In addition, the current weather may influence the achievable level of accuracy, as well as the devices used as tracking devices and / or environmental modeling devices. In addition, the area in which the surrounding environment is detected has an impact, because, for example, a flat parking lot contains far fewer targets that can be tracked by an environmental modeling device than, for example, a parking lot where one or more large working tools are stored (temporarily). In addition, the higher the level of accuracy may be, the better the error sources of the acquired data are modeled.

[0149] On the validity of the position and orientation determined by the machine in the work site, in the case where the machine is preferably substantially continuously moving, it may be affected by the acquisition of data by at least one environmental modeling device and at least one tracking device sufficiently frequently, so that the determined position and orientation will not be based on very old data. If the determined position and orientation of the machine is based on data acquired by a tracking device TA and / or an environmental modeling device EM arranged in a specific machine or work site, and the machine is additionally provided with its own sensors, the validity of the data acquired by the corresponding device can be improved by the data provided by the sensors in the machine. This allows the work task to continue to be performed in the case where the position and orientation of the machine are determined using data that is blocked or considered unreliable acquired by at least one tracking device TA and / or at least one environmental modeling device EM. In this case, the data acquired by the device does not necessarily need to be very recent. However, if the determined position and orientation of the machine is based on data acquired by a tracking and / or environmental modeling device arranged in another machine, the data acquired by the device in the other machine should be very recent, that is, the timestamp of the acquired data should be very recent, preferably as recent as possible, because the machine whose position and orientation is to be determined is generally not able to verify or improve the validity of the acquired data, because it is not possible to detect any changes in the position or orientation of the other machine to which the device for acquiring data is attached. On the other hand, a machine that can acquire data from devices disposed in another machine can also receive data related to these devices, for example, which can indicate the accuracy and / or validity of the data.

[0150] According to an embodiment, data related to at least one tracking device TA include at least one of the following: the position of the tracking marker point MP relative to the tracking device TA, the position of the tracking reference point RP relative to the tracking device TA, the inclination of the tracking device TA, the direction of the tracking device TA, the stability of the tracking device TA, the position and orientation of the tracking device TA in at least one of the mechanical coordinate system MCS, the work site coordinate system WCS or the world coordinate system WLCS, or at least one of the accuracy level or validity of at least one of the previous ones. Fig.16The data related to at least one tracking device TA according to the present embodiment are schematically shown. The data related to at least one tracking device TA may also include other information not specifically disclosed herein. Therefore, the data related to the tracking device TA may also include the position and / or orientation data of the tracking device TA itself, which can be received without the operator's knowledge and can be used by the machine to determine its own position and / or orientation. As described above, the data discussed herein may be provided by a device and / or sensor arranged in, for example, the same machine to which each tracking device TA is attached, or the data discussed herein may be provided by a device and / or sensor arranged in, for example, another machine to which each tracking device TA is attached. Alternatively, the data discussed herein may be provided by a combination of the aforementioned data, such that if a device or sensor provides at least one information, it may be added to the data related to the tracking device TA, or, if another device or sensor provides another at least one information, it may also be added to the data related to the tracking device TA. Furthermore, if two or more devices or sensors provide the same information, e.g. the inclination of the tracking device TA, the two or more information may be combined mathematically, e.g. by averaging or weighted averaging, or it may be selected which of the two or more information to add to the data relating to the tracking device TA, e.g. by selecting one that is considered more accurate and / or more efficient.

[0151] The position of the tracking marker point MP relative to the tracking device TA and the position of the tracking reference point RP relative to the tracking device TA may be determined as disclosed above in connection with other embodiments in this description.

[0152] The inclination of the tracking device TA, the heading of the tracking device TA and the stability of the tracking device TA may be determined by sensors arranged in the tracking device TA as disclosed above in this specification, or alternatively, by sensors arranged in the machine if the position and orientation of the tracking device TA in the machine coordinate system MCS are known. The position and orientation of the tracking device TA may be determined by three-dimensional coordinates and inclination and heading angles (such as roll, pitch and yaw) in the work site coordinate system WCS and / or three-dimensional coordinates and inclination and heading angles (such as roll, pitch and yaw) in the above-mentioned machine coordinate system MCS. Alternatively, or in addition, the position and orientation of the tracking device TA may be determined by three-dimensional coordinates and inclination and heading angles (such as roll, pitch and yaw) in a world coordinate system WLCS, for example a satellite-based positioning system GNSS.

[0153] The level of accuracy or validity of data related to at least one tracking device TA may be affected by the number of tracking devices used, by the number of reference points and marker points used and their distances from the tracking devices, and by other factors as described above, such as the stability of the tracking device TA and the acquisition of data frequently enough by at least one tracking device TA and possible sensors in the tracking device TA and / or in the machinery.

[0154] According to an embodiment, the data related to at least one environment modeling device is at least one of the following: spatial data, the position of a tracking marker point MP relative to the environment modeling device EM, the position of a tracking reference point RP relative to the environment modeling device EM, the inclination of the environment modeling device EM, the heading of the environment modeling device EM, the stability of the environment modeling device EM, the position and orientation of the environment modeling device EM in at least one of the mechanical coordinate system MCS, the work site coordinate system WCS or the world coordinate system WLCS, or at least one of the accuracy level or validity of at least one of the previous ones. Fig.17 Schematically illustrated are data related to at least one environment modeling device EM according to the present embodiment. The data related to at least one environment modeling device EM may also include other information not specifically disclosed herein. Therefore, the data related to the environment modeling device EM may also include the position and / or orientation data of the environment modeling device EM itself, which data may be received without the operator's knowledge and may be used by the machine to determine its own position and / or orientation. As described above, the data discussed herein may be provided by a device and / or sensor arranged in, for example, the same machine to which each environment modeling device is attached, or the data discussed herein may be provided by a device and / or sensor arranged in, for example, another machine to which each environment modeling device EM is attached. Alternatively, the data discussed herein may be provided by a combination of the aforementioned data, such that if a device or sensor provides at least one information, it may be added to the data related to the environment modeling device EM, or, if another device or sensor provides another at least one information, it may also be added to the data related to the environment modeling device EM. Furthermore, if two or more devices or sensors provide the same information, for example the tilt of the environment modeling device EM, these two or more information may be combined mathematically, for example by averaging or weighted averaging, or it may be selected which of the two or more information is to be added to the data relating to the environment modeling device EM, for example by selecting one which is considered to be more accurate and / or more efficient.

[0155] The spatial data tracked and / or detected by the environmental modeling device EM may be raw data or pre-processed data, which may relate to the entire work site 13 or a specific portion thereof. As described earlier in this specification, the spatial data includes, for example, data related to the entire work site 13, or data related to a specific portion of the work site 13, which may be the same portion in which the discussed machine will actively operate or another portion of the work site 13. The spatial data may include surface contours of the work site or a specific portion thereof, permanent or temporary objects such as buildings or storage rooms, rocks, rock masses, trees, work machinery or auxiliary work machinery, materials used in the work site or a specific portion thereof, etc. These objects may include visible and invisible objects, such as underground pipelines or power lines. The spatial data may also include environmental conditions of the work site or a specific portion thereof, such as temperature, humidity, and rainfall depth, for example, hourly or daily. The spatial data related to the work site or a specific portion thereof may also include other information related to the work site or a specific portion thereof that can be detected by at least one available device.

[0156] The position of the tracking marker MP relative to the environment modeling device EM and the position of the tracking reference point RP relative to the environment modeling device EM can be determined in a similar manner to the position of the tracking marker MP and the tracking reference point RP relative to the tracking device TA. In this regard, the above specifications in this specification regarding the tracking device TA also apply to the environment modeling device EM.

[0157] The inclination angle of the environment modeling device EM, the orientation of the environment modeling device EM and the stability of the environment modeling device EM can be determined by sensors arranged in the environment modeling device EM. Figure 7 The specifications and embodiments related to the tracking device TA and the sensors therein also apply to the environment modeling device EM. Alternatively, if the position and orientation of the environment modeling device EM in the machine coordinate system MCS are known, the inclination of the environment modeling device EM, the heading of the environment modeling device EM and the stability of the environment modeling device EM can be determined by sensors arranged in the machine.

[0158] Similar to the tracking device TA above, the position and orientation of the environment modeling device EM may be determined by three-dimensional coordinates and inclination and heading angles (such as roll, pitch and yaw) in the work site coordinate system WCS and / or three-dimensional coordinates and inclination and heading angles (such as roll, pitch and yaw) in the machine coordinate system MCS. Alternatively, or in addition, the position and orientation of the environment modeling device EM may be determined by three-dimensional coordinates and inclination and heading angles (such as roll, pitch and yaw) in the world coordinate system WLCS.

[0159] The information related to the environment modeling device EM can be used in a variety of ways to model the work site 13 to determine the situational awareness in the work site 13. In relation to the specific work task to be performed, the situational awareness obtained can, for example, include information about the initial state of the work site 13 or a part thereof before the start of the work task, as well as information about the final result after the performance of the specific work task. Additionally, the situational awareness can include information about intermediate stages of the work site 13 or a part thereof during the performance of the work task, as well as obtaining information related to deviations that occurred during the performance of the work task, such as information related to objects that were initially invisible but appeared or became visible during the performance of the work task, such as underground rocks or rock bodies that became visible during excavation work. The disclosed solution for determining situational awareness allows the measurement or modeling of objects that have appeared during the performance of the current work task by the provided device, thereby allowing possible planning of future work tasks, such as the removal of rocks or rock bodies.

[0160] The accuracy level or validity of the data related to the at least one environment modeling device TA may be influenced by the number of environment modeling devices used, by the number of reference points and markers used and the area of ​​the surrounding environment detected, and by their distances from the environment modeling device, and by other factors as described above, such as the stability of the environment modeling device EM and the fact that the at least one environment modeling device EM and possible sensors in and / or in the machinery of the environment modeling device EM acquire data frequently enough. Therefore, the above description about the accuracy level or validity of the data related to the at least one tracking device TA also applies to the accuracy level or validity of the data related to the at least one environment modeling device EM.

[0161] The accuracy of the data acquired by the at least one environment modeling device EM and the accuracy of the data acquired by the at least one tracking device TA can also be improved by determining average information of the acquired data. This is particularly useful in the case where the at least one tracking device TA and / or the at least one environment modeling device EM are arranged in a machine and remain in a single position for a long time, whereby the average information of several single data information acquired at different times can be determined as long as the machine remains in the same position during the acquisition of said single data information.

[0162] In case at least one tracking device TA and / or at least one environment modeling device EM is arranged in a mobile machine, the determination of average information may also be applied as long as at least one point or object to be tracked or monitored remains the same during acquisition of said single data information.

[0163] Furthermore, a weighted average can be applied for improving the accuracy of the acquired data. According to this embodiment, data acquired by a device that is held closer to the object to be monitored can have a higher weight than data acquired by a device that is held further away from the object to be monitored. This can be used to improve the accuracy of the acquired data, especially in the case of mechanical movement.

[0164] Accurate determination of situational awareness may also require correct synchronization of the determined position and orientation of the machine with the data modeling the work site 13 or a portion thereof and providing data related to situational awareness in the work site 13. In the case of a machine remaining in one position for a long period of time, very precise synchronization is not important for accurate determination of situational awareness because the machine is not moving. However, in the case of a mobile machine, the importance of synchronization increases to ensure correct compliance between the determined position and orientation of the machine and the data modeling of the work site 13 or a portion thereof. Therefore, in particular for mobile machines, the required data can be acquired even within a time period of milliseconds so that the determined position and orientation of the machine is correctly synchronized with the data modeling of the work site 13 or a portion thereof. In this case, there is also no ultimate requirement for real-time between the acquisition of data determining the position and orientation of the machine and the modeling of the work site 13 or a portion thereof, as long as they can be synchronized with each other.

[0165] If the at least one tracking device TA and the at least one environment modeling device EM are the same device, the acquired data for determining the position and orientation of the machine and the acquired data for modeling the work site 13 are automatically synchronized.

[0166] According to an embodiment, the spatial data includes at least one of the following: graphical data, point cloud data, or data with an implicit or explicit reference to a position relative to at least one of the work site 13 or the earth. According to this embodiment, data describing spatial data, in particular spatial data related to physical objects in the work site 13, can be represented by graphical data (i.e., by using pictures) and / or by point cloud data, whereby the physical objects can be described by point clouds. Although the content of specific data items of spatial data and their representation methods are different, each specific data item can be combined with an implicit or explicit reference to a position relative to the work site 13 or the earth, thereby allowing the information of the specific data item to be assigned at a specific position on the work site. The explicit reference can be, for example, a position in a work site coordinate system WCS or in a world coordinate system WLCS, while the implicit reference can be, for example, a position of a point cloud seen by an environment modeling device in an environment modeling device coordinate system, whose position and orientation are known in the work site coordinate system WCS or in the world coordinate system WLCS.

[0167] According to an embodiment, the method further comprises: receiving, by at least one environmental modeling unit EMU, an indication of material delivery, a material delivery base determined by first data related to at least one environmental modeling device EM covering the area where the material is to be placed, and a material delivery completion determined by second data related to at least one environmental modeling device EM covering the area where the material is to be placed, and at least partially saving, by at least one environmental modeling unit EMU, the data on the indication of material delivery, the material delivery base, and the material delivery completion as the delivered material. It should be noted that the first data and the second data may be determined by different at least one environmental modeling device EM.

[0168] The indication of the material delivery may include information or data on the actual delivery of the material, i.e. the date and time at which the material delivery was made, and a product description of the delivered material. The indication of the material delivery may include other information related to the content of the delivered material, such as the type or grade of the delivered material, and / or the quantity and / or weight of the delivered material, and / or the volume and / or color of the delivered material, as separate information or part of the product description of the delivered material.

[0169] In response to an area of ​​the work site 13 or a specific part thereof where material is to be delivered, at least one environmental modeling device EM covering the area is configured to determine a material delivery base at the area where the material to be delivered is to be placed. In other words, according to this embodiment, at least one environmental modeling device EM is configured to determine, for example, the surface contour of the area of ​​the work site 13 where the material to be delivered is to be placed. This allows, later, when the delivered material is moved to another place (for example when used for its final use), to control the work machine so that substantially all the delivered material (but preferably not the material of the delivery base) is removed from the material delivery base where the delivered material is placed, so that the material delivery base remains substantially the same as before the material was delivered. The information related to the material delivery base provides first data related to at least one environmental modeling device EM covering the area where the material is to be placed.

[0170] In response to the material having been delivered to the work site 13 or a specific part thereof, the at least one environment modeling device EM covering the area is configured to determine information related to the completion of the material delivery, i.e. information describing the completion or end of the material delivery, such as a pile of rubble. This allows, for example, the planning of a later work phase, such as scheduling the laying of rubble, to control the corresponding work machine. The information related to the completion of the material delivery provides second data related to the at least one environment modeling device EM covering the area where the material is to be placed.

[0171] At least part of the data on the indication of the material delivery, the material delivery base and the completion of the material delivery is saved by at least one environmental modeling unit EMU as a delivery material describing the completion or end of the material delivery and its characteristics.

[0172] According to an embodiment, the method further comprises: receiving, by at least one environmental modeling unit EMU, one or more indications relating to area data of the work site, i.e. data relating to an area of ​​a part of the work site and comprising information relating to a work period and / or a work phase in a corresponding area of, for example, a part of the work site. Additionally, the at least one environmental modeling unit EMU receives data relating to at least one environmental modeling device EM and, by taking into account one or more indications, which are derived from the data relating to the at least one environmental modeling device EM, are geo-referenced spatial data of the corresponding area and at least partially saves the geo-referenced spatial data into, for example, a model.

[0173] According to this embodiment, at least one environmental modeling unit EMU is configured to receive one or more indications related to work periods and / or work stages of corresponding areas of multiple parts of the work site 13, for example from an application interface used for work site management. These indications may include information related to work periods or operations that have been performed or are currently being performed in the work site 13 or at a specific part thereof. Alternatively, or in addition, these indications may include information related to the stage of the work site 13 or a specific part thereof, that is, the progress level of the work site 13 or a specific part thereof. The progress of work at different parts of the work site 13 may be different. In addition, at least one environmental modeling unit EMU is configured to receive data related to at least one environmental modeling device EM, data related to at least one environmental modeling device EM that has been discussed in more detail above in this specification.

[0174] Furthermore, the at least one environmental modeling unit EMU is configured to: consider one or more indications and derive geo-referenced spatial data of corresponding areas of a plurality of parts of the work site 13 from data related to the at least one environmental modeling device EM, and to save the geo-referenced spatial data at least partially in a model or in a geo-referenced spatial data model. In addition to utilizing a possible storage unit in the position determination unit PDU, the at least one environmental modeling unit EMU may include at least one storage unit to at least temporarily store the saved data. For example, the data and / or the model may be saved to a work site computer and / or a cloud service and / or any other memory, database or data structure suitable for storing data and / or models.

[0175] Geo-referenced spatial data refers to spatial data (which has been discussed in more detail above), but the spatial data is further combined with more specific or accurate location information to determine or establish the location of the spatial data item or object in the work site or a specific portion thereof. An important part of geo-referenced spatial data is as-built data, which describes the progress stage or level of work in the work site or a specific portion thereof.

[0176] According to an embodiment, the step of at least partially saving georeferenced spatial data also includes: determining the area whose georeferenced spatial data is to be saved based at least in part on data received from at least one environmental modeling device, and saving the georeferenced spatial data of the area determined to be saved. The area of ​​georeferenced spatial data refers to an area in the work site in this article. If new georeferenced spatial data is obtained from a parking space area of ​​the work site to be saved in the model, and an additional excavator bucket is stored in the area, the point cloud describing the additional excavator bucket will not be saved as the gravel level of the area, but the area next to the bucket can be saved as a point cloud, if it is known from the history of the area that the work period is to fill the gravel to the level indicated in the BIM model, then the point cloud describes the gravel level. Therefore, according to this embodiment, not all georeferenced spatial data are to be saved, but for example, the area in the retrieved data is to be saved, such as the operator's own work area (such as the part of the roadbed where the operator is working). In addition, the environmental modeling unit EMU can have an indication related to the adjacent part of the roadbed, where the operator's colleagues are working. A colleague or work site manager may be interested in the geo-referenced spatial data collected by the operator about the section of the roadbed that the colleague is working on, so the operator's environmental modeling device may also save geo-referenced spatial data reaching adjacent sections of the roadbed, but may not save areas beyond its roadbed combination work area. In addition, such geo-referenced spatial data related to projects or objects may also be saved, which may have an impact on determining the position and orientation of the work machinery to advance the progress of the work site. Therefore, for example, if the point cloud describing the personnel or vehicles temporarily staying in the work site does not have any impact on determining the position and orientation of the work machinery considering the progress of the work site, these point clouds will not necessarily be saved. The choice not to save the data can be provided or assisted by the operator of the machine, or the choice not to save the data can be fully automatic using, for example, a neural network application. If necessary, other applications, such as machine control level applications, can be used in the control of the machine, for example to avoid any collision between the work machinery and the temporary objects of the type. For example, the detected temporary objects can be collected into their own database or data structure. In addition, for example, tools and / or materials located in or related to the work site can also be collected into their own database or data structure.

[0177] According to an embodiment, the step of determining an area of ​​geo-referenced spatial data to be saved based at least in part on data received from at least one environmental modeling device includes: detecting an area in which the geo-referenced spatial data of at least one environmental modeling device is unobstructed, and treating the area as an unobstructed area. In addition, comparing the current accuracy of the determined position and orientation of the machine in the work site with the accuracy of the determined position and orientation of the machine in the work site at the time of the previously saved geo-referenced spatial data, and whether the current accuracy is higher than approximately the same accuracy, updating the geo-referenced spatial data saved in the unobstructed area. Updating data in this context includes at least one of the following: replacing, overwriting, adjusting, adding or averaging data using known mathematical methods, such as, for example, averaging or weighted averaging.

[0178] In this embodiment, unobstructed areas (i.e. areas providing information about the actual phase of the work site or a specific part thereof) in the geo-referenced spatial data provided by the at least one environment modeling device EM are determined, for example, areas where no temporary obstacles appear in the captured view of the at least one environment modeling device EM. The determination is made by the position determination unit PDU based on these unobstructed areas in the geo-referenced spatial data, the current or present position and orientation of the machine in the work site 13, and the accuracy of the position and orientation of the machine. Furthermore, the accuracy related to the current position and orientation of the machine in the work site 13 is compared by the position determination unit PDU with the accuracy of the position and orientation of the machine with respect to the previously saved or stored geo-referenced spatial data. In response to the above almost identical accuracy related to the current position and orientation of the machine (i.e. taking into account the margin or error in determining the accuracy), and at least the same or even better accuracy than the accuracy of the position and orientation of the machine with respect to the previously saved geo-referenced spatial data, the saved geo-referenced spatial data are updated, i.e. replaced, overwritten, adjusted, added or averaged using known mathematical methods (e.g. average or weighted average in unobstructed areas). Furthermore, when the accuracy criteria are met, a threshold level for each accuracy level may be further determined to determine how the geo-referenced spatial data should be updated in each case. For example, the threshold level may be a margin error in the determined accuracy level. For example, if the actual position of the tool in the work site coordinate system is determined to be within 20 mm, the threshold may be 20 mm, 40 mm or even 80 mm. The use of a threshold ensures that very slight or negligible changes, i.e. changes below the threshold in an area of ​​the geo-referenced spatial data, do not initiate a rewrite or replacement of previously saved geo-referenced spatial data, but may initiate, for example, averaging or weighted averaging of the geo-referenced spatial data with previously saved geo-referenced spatial data. If the change is above the determined threshold, the update may be a replacement or rewrite of the geo-referenced spatial data, since in this case it should be assumed that a change has occurred in the environment.

[0179] According to an embodiment, at least one position determination unit that determines the position and orientation of a machine in a work site parses data about at least one of: a tracking device, an environmental modeling device, an object, or another machine; and transmits the parsed data as at least one of: a portion of data related to the corresponding tracking device, environmental modeling device, object, or another machine, or data capable of being received by at least one position determination unit.

[0180] According to this embodiment, at least one position determination unit PDU that determines the position and orientation of a machine in a work site 13 can parse, i.e., for example, determine data related to at least one of the following in combination with its own measurement data: for example, a tracking device TA, an environmental modeling device EM, an object or another machine. The object can be an object to be avoided, such as an obstacle, or an object to be approached or of interest, such as a material to be used in the work. After this parsing, the parsed data is transmitted to, for example, a work site computer and / or a cloud service and / or a memory, a database or a data structure, which is suitable for saving the data as at least one of the following: part of the data related to the corresponding tracking device TA, the environmental modeling device EM, the object or another machine, and / or data that can be received by at least one position determination unit PDU. The accuracy and / or validity of such parsed data depends on the accuracy and validity of each sensor and / or device related to providing such data. For example, if the position and orientation of a machine that interprets the data is determined by an environment modeling device EM arranged on the machine in combination with data received from sensors mounted on the machine, the accuracy and / or validity of the interpreted data depends on the accuracy and validity of the combination of data acquired to determine the position and orientation of the machine, and the accuracy and validity of any other data related to the interpretation. Here, the accuracy of the data acquired by the environment modeling device EM depends, for example, on whether the machine is stable when tracking, and whether the distance of the tracked surroundings or tracked reference points or other trackable markers is close or far, and how many of them are tracked and how accurately their positions are determined.

[0181] According to an embodiment, the tracking device is further initialized by determining the position and orientation of the tracking device in at least one of a machine coordinate system or a work site coordinate system if the tracking device is set on at least one of the machine or another machine, and determining the position and orientation of the tracking device in the work site coordinate system if the tracking device is set outside of any machine.

[0182] The initialization of the tracking device TA has been considered in this specification. In addition, according to the present embodiment, in the initialization of the tracking device TA, the position and orientation of the tracking device TA can be determined in the machine coordinate system MCS and / or in the work site coordinate system WCS. If the tracking device TA is arranged in a machine or another machine, the position and orientation of the tracking device TA can be determined either in the machine coordinate system MCS or in the work site coordinate system WCS or in both, depending on the location of the tracking device and the location to which it is attached. If the tracking device uses an external positioning system such as GNSS, it will be determined by itself in the work site coordinate system WCS and determined in the coordinate system of the machine to which it is attached. On the other hand, if the tracking device TA does not use an external positioning system and it is attached to a machine (i.e., a machine or another machine), the tracking device is at least determined in the coordinate system of the machine to which it is attached, and if it is used as a tracking device by another machine, it is determined in the work site coordinate system WCS. If the tracking device TA is arranged outside any machine, the position and orientation of the tracking device TA should be determined at least in the work site coordinate system WCS.

[0183] According to an embodiment, the environmental modeling device is further initialized by determining the position and orientation of the environmental modeling device in at least one of a machine coordinate system or a work site coordinate system if the environmental modeling device is set on at least one of the machine or another machine, and determining the position and orientation of the environmental modeling device in the work site coordinate system if the environmental modeling device is set outside of any machine.

[0184] The above description related to the initialization of the tracking device TA is also correspondingly applicable to the initialization of the environment modeling device EM. In addition, according to the present embodiment, in the initialization of the environment modeling device EM, the position and direction of the environment modeling device EM can be determined in the machine coordinate system MCS and / or the work site coordinate system WCS. If the environment modeling device EM is arranged in a machine or another machine, it depends on the environment modeling device and the position to which it is attached, and the position and direction of the environment modeling device EM can be determined in the machine coordinate system MCS or in the work site coordinate system WCS or both. If the environment modeling device uses an external positioning system such as GNSS, it will be determined in the work site coordinate system WCS by itself and determined in the coordinate system of the machine to which it is attached. On the other hand, if the environment modeling device EM does not use an external positioning system and it is attached to a machine (i.e., a machine or another machine), the environment modeling device is at least determined in the coordinate system of the machine to which it is attached, and if it is used by another machine, it is determined in the work site coordinate system WCS. If the environment modelling equipment EM is arranged outside any machinery, the position and orientation of the environment modelling equipment EM shall be determined at least in the work site coordinate system WCS.

[0185] According to an embodiment, the determined situational awareness is at least one of spatial data, geo-referenced spatial data, regional work period and / or work stage data, completion data, at least one of the position, orientation, driving direction or alternative driving direction of any machine in the work site, at least one of the position, orientation or driving direction or alternative driving direction of at least one of the machines, obstacles or objects that are stationary or moving, to be avoided or of interest, or at least one of the surrounding environment.

[0186] Fig.18 Schematically disclosed are some data that depict or provide situational awareness in the worksite 13. The determined situational awareness may contain or include many different pieces of data or information, depending on the worksite or a particular portion thereof.

[0187] Some examples of data or information segments have been discussed above in this specification, which data or information segments may form: at least partial spatial data, geo-referenced spatial data, regional data, as-built data; or at least one of the position, direction, direction of travel, or alternate direction of travel of any machinery in the work site 13.

[0188] In addition to (or in addition to) the data disclosed in the preceding paragraphs, data or information segments relating to at least one of the position, orientation or direction of travel or alternate direction of travel or surroundings of a stationary or moving machine, obstacle or object may be used to determine or indicate situational awareness in the work site 13. The disclosed machine, obstacle or object may be either stationary or moving. Obstacles herein refer to objects that may be either stationary or moving, but in any case are objects that are actively avoided, for example by going around or out of the way. Objects herein may in turn refer to objects to be avoided, such as the obstacles above, or to objects of interest that may be intentionally approached, such as materials to be used in the work performed by the machine.

[0189] According to an embodiment, at least one tracking device tracks the position of at least one of the following relative to the tracking device: at least one reference point in the work site, at least one marking point attached to the machine, or any other trackable marker attached to at least one of the machine, obstacle or object; and at least one environmental modeling device tracks the position of at least one of the following relative to the at least one environmental modeling device: at least one reference point in the work site, at least one marking point attached to the machine, any other trackable marker attached to at least one of the machine, obstacle or object, or spatial data related to the work site.

[0190] According to this embodiment, at least one tracking device TA is configured to track the position of at least one reference point RP in the work site 13 and / or at least one marker point MP attached to the machine relative to the tracking device TA. Instead of at least one of these, or in addition to at least one of these, at least one tracking device TA is configured to track the position of any other trackable marker attached to the machine and / or obstacle and / or object. For example, any other trackable marker can be a marker attached to a large stone, a solid rock or a large tree or any fairly stationary object, which can be used as an unofficial reference point, which can be used as an additional trackable marker to assist in determining the position and orientation of the machine. In particular, it can be used in areas with fewer trackable reference points. The machine can resolve the exact position of such a marker in the work site coordinate system WCS and / or the world coordinate system WLCS, and after resolution, transmit it to: for example, a work site computer and / or a cloud service and / or a memory, a database or a data structure, which is suitable for saving the data as part of the data related to the trackable marker and / or data that can be received by at least one position determination unit PDU. Such data should also contain information about the accuracy and / or validity of the data, because, for example, the rock may have moved a little due to some work task close to the rock, or the worker may have changed the trackable marker to another location. In the latter case, the worker who moved the trackable marker should also remove the resolved position saved for the trackable marker from the location where it was saved. The tracking provided by the tracking device TA operates as disclosed in the above embodiments.

[0191] Furthermore, according to the present embodiment, at least one environment modeling device EM is configured to track the position of at least one reference point RP in the work site 13 and / or at least one marker point MP attached to the machine relative to the at least one environment modeling device EM. Instead of at least one of these (or in addition to at least one of these), at least one environment modeling device EM is configured to track the position of any other trackable marker attached to the machine and / or obstacles and / or objects, and / or spatial data related to the work site 13, wherein the spatial data related to the work site 13 can be, for example, natural landmarks such as trees and / or boulders and / or solid rocks and / or small stones and / or other landmarks such as immovable objects and / or tools and / or buildings and / or warehouses and / or stumps. It should be noted that the environment modeling device EM can use a number of landmarks to keep track of the position and orientation of the machine, which means that if the position and orientation are determined at one instant, the position and orientation are determined by determining the extent to which the position and orientation of the machine changes after the instant at which the position and orientation are determined relative to these landmarks, and the environment modeling device EM can use these landmarks to determine the position and orientation of the machine after the position and orientation are determined. The environment modeling device EM can then continue such tracking of the position and orientation of the machine without having to detect such reference points and / or other trackable marking points whose positions are known in the model. Naturally, when such tracking is prolonged without detecting reference points and other trackable marking points, the accuracy of the determined position and orientation of the machine decreases. The operator can be informed of such decrease. The tracking performed by the at least one environment modeling device EM can operate similarly to the tracking performed by the at least one tracking device. Therefore, in this respect, the above specifications regarding the tracking device TA also apply to the environment modeling device EM.

[0192] According to an embodiment, the method for determining situational awareness in a work site further includes: determining a minimum level of accuracy for a determined position and orientation of a machine in the work site, determining a threshold level above the minimum level of accuracy, and wherein, if the level of accuracy is below the threshold level and if the work task being performed can be performed without moving the lower carriage, disabling a control device for moving a lower carriage of the machine.

[0193] The present embodiment relates in particular to a machine having a movable bracket, the movable bracket comprising a lower bracket and an upper bracket capable of moving relative to each other, the lower bracket and the upper bracket rotating relative to each other as in an excavator. According to the present embodiment, a minimum level of accuracy of the determined position and orientation of the machine is determined, the minimum level of accuracy being required to allow all possible operations of the working machine to fully operate. If the minimum level of accuracy of the determined position and orientation of the machine is not reached, the machine is allowed to operate with limited operation. Therefore, a threshold level of accuracy of the determined position and orientation of the machine is additionally determined, and in response to the level of accuracy of the determined position and orientation of the machine being below the threshold level, the operation of the machine may be limited so that the machine can continue its operation but is not allowed to move from the current position in the work site 13. If the machine is an excavator 1, this means that the lower bracket 2a of the excavator 1 is not allowed to move, but the excavator 1 can continue to operate as long as it does not need to move from the current position.

[0194] According to an embodiment, data relating to at least one of the at least one tracking device or at least one of the at least one environmental modeling devices comprises at least one of: data acquired by the respective device, data from a sensor mounted on the respective device, data from a sensor mounted on an attachment point of the device, data parsed by any position determination unit or at least one of the devices through at least one of tracking the respective device or as a result of any calculation relating to at least one of a previous level of accuracy or validity of the respective device or at least one.

[0195] According to the present embodiment, the data related to at least one tracking device TA and / or at least one environment modeling device EM include data acquired by the respective device and / or data from possible sensors mounted on the respective device, which sensors have been discussed in more detail in the above description. Alternatively or additionally, the data related to at least one tracking device TA and / or at least one environment modeling device EM may include data from sensors mounted on attachment points of the devices, whereby the sensors may, for example, provide information about the position and / or orientation and / or inclination and / or orientation of the respective device. Alternatively or additionally, the data related to at least one tracking device TA and / or at least one environment modeling device EM may include data parsed by any position determination unit and / or any device by tracking the respective device and / or data as a result of any calculation related to the accuracy level and / or validity of the respective device and / or at least one of the aforementioned data. Thus, the data related to at least one tracking device TA and / or at least one environment modeling device EM may be retrieved in many different ways, such as by a tracking operation provided by the respective tracking device, parsing data in at least one position determination unit, or by a calculation operation, for example, provided by at least one position determination unit, also taking into account the accuracy level and / or validity of the data.

[0196] According to an embodiment, if at least one tracking device is arranged on a work site and if it comprises a tracking device for tracking the position of the tracking device using one or more GNSS antennas, the tracking device further comprises at least one of the following as the tracking device: a camera, a stereo camera, a lidar, a radar or a speedometer.

[0197] According to this embodiment, if the tracking device TA is arranged on the work site 13 and if it comprises a tracking device TD for tracking the position of the tracking device TA using one or more GNSS antennas, it is also provided with at least one camera and / or at least one stereo camera and / or at least one lidar and / or at least one radar and / or at least one velocimeter as further tracking device TD for tracking the position and orientation of the machine in the work site 13 and at the same time for tracking the position and orientation of other objects or obstacles in the work site 13. In particular, radar can also be used to track invisible objects such as underground buildings.

[0198] According to an embodiment, in order to determine situational awareness in a work site, at least one position determination unit is configured to determine the position and orientation of the machine in the work site additionally at least in part based on data received from one or more sensors mounted on the machine and / or another machine; wherein the sensors include at least one of the following: the position, orientation, inclination, heading or travel distance of the machine and / or the other machine. According to this embodiment, the determination of the position and orientation of the machine can be additionally based at least in part on sensor data describing the position and / or orientation and / or inclination and / or heading and / or travel distance of the machine and / or the other machine (if data related to the other machine are used to determine the position and orientation of the relevant machine) whose position and orientation is to be determined.

[0199] According to an embodiment for determining situational awareness in a work site, the machine is an excavator and at least one position determination unit PDU is additionally configured to determine the position and orientation of the machine in the work site based at least in part on data received from one or more sensors mounted on an upper carriage of the machine and / or another machine, wherein the sensors include the position and / or orientation and / or inclination and / or heading of the upper carriage of the machine and / or another machine. According to this embodiment, the machine is an excavator 1 and the determination of the position and orientation of the excavator 1 can additionally be based at least in part on sensor data describing the position and / or orientation and / or inclination and / or heading of the upper carriage 2b of the excavator 1 and / or the other excavator 1 (if data related to the other excavator 1 is used to determine the position and orientation of the relevant excavator 1) whose position and orientation is to be determined.

[0200] An excavator 1 is an example of a mobile earth-moving machine that can be used in conjunction with a solution for determining the position and orientation of the machine in a work site and for determining situational awareness in a work site. In addition to excavators like this, the solution disclosed herein can also be used, for example, in a mobile crane, wherein the mobile crane includes a bracket member arranged to rotate relative to the rest of the mobile crane, and wherein the rotatable bracket member includes a lifting boom and a hook for setting the working tool of the crane at the distal end of the boom. The solution for determining the position and orientation of the machine in a work site and for situational awareness in a work site is basically similar to that in a mobile crane. In which the disclosed solution can be utilized, in addition to excavators and mobile cranes, other machines are for example bulldozers, wheel loaders, rollers, backhoes, dump trucks, transporters, harvesters, etc.

[0201] It will be clear to a person skilled in the art that, as technology advances, the concept of the invention can be implemented in various ways.The invention and its embodiments are not limited to the examples described above but may vary within the scope of the claims.

Claims

1. A method for determining situational awareness in a work site, characterized in that The method comprises at least one environment modeling device is arranged at least one of: on a machine or on the outside of the machine, the machine being used to detect at least a surrounding area as spatial data of the work site or spatial data related to the work site, wherein the spatial data is at least a surface contour of the ground; Disposing at least one tracking device at least one of: on a machine for tracking at least at least one reference point or external to a machine for tracking at least at least one reference point and at least one marker point, wherein the at least one reference point is in the work site and the at least one marker point is attached to the machine; acquiring data by the at least one tracking device by tracking the positions of reference points and marker points relative to the tracking device; acquiring data by the at least one environmental modeling device by tracking spatial data of or relating to the work site relative to the at least one environmental modeling device; Received by at least one location determining unit data related to the at least one tracking device, comprising at least data acquired by the at least one tracking device, and data related to the at least one environmental modeling device, comprising at least data acquired by the environmental modeling device; and A position and orientation of the machine within the work site is determined by the at least one position determining unit based at least in part on the received data.

2. The method according to claim 1, characterized in that The method further comprises At least one of: a direction of travel or an alternate direction of travel of the machine within the work site is determined by the at least one position determination unit based at least in part on the received data.

3. The method according to claim 1, characterized in that The method also includes determining at least one of: a level of accuracy or validity of the determined position and orientation of the machine in the work site.

4. The method according to claim 1, characterized in that: The data related to the at least one tracking device also includes at least one of the following: the inclination of the tracking device, the heading of the tracking device, the stability of the tracking device, the position and orientation of the tracking device in at least one of a machine coordinate system, a work site coordinate system, or a world coordinate system, or at least one of the accuracy level or validity of at least one of the previous ones.

5. The method according to claim 1, characterized in that The data related to the at least one environmental modeling device is also at least one of the following: the position of a tracking marker point relative to the environmental modeling device, the position of a tracking reference point relative to the environmental modeling device, the inclination of the environmental modeling device, the heading of the environmental modeling device, the stability of the environmental modeling device, the position and orientation of the environmental modeling device in at least one of a machine coordinate system, a work site coordinate system or a world coordinate system, or at least one of the accuracy level or validity of at least one of the previous ones.

6. The method according to claim 5, characterized in that The spatial data includes at least one of: graphical data, point cloud data, or data implicitly or explicitly referencing a location relative to at least one of: the work site or the earth.

7. The method according to claim 1, characterized in that The method further comprises: received by at least one environment modeling unit Instructions for material delivery; a material distribution base determined by first data associated with at least one environmental modeling device covering an area where the material is to be placed; and the material delivery being complete as determined by second data associated with at least one environmental modeling device covering the area where the material is to be placed; and Data regarding the indication of the material delivery, the material delivery base, and the completion of the material delivery are at least partially stored by the at least one environmental modeling unit as the delivered material.

8. The method according to claim 1, characterized in that The method further comprises: received by the at least one environment modeling unit one or more indications related to at least one of a working period or a working stage of a corresponding area; and data related to the at least one environmental modeling device; and wherein By considering the one or more indications deriving geo-referenced spatial data for the corresponding area from data associated with the at least one environmental modeling device; and The geo-referenced spatial data is at least partially saved.

9. The method according to claim 8, characterized in that The step of at least partially saving the geo-referenced spatial data further comprises: determining an area for which geo-referenced spatial data is to be saved based at least in part on data received from the at least one environmental modeling device, and Save determines the area of ​​georeferenced spatial data to be saved.

10. The method according to claim 9, characterized in that The step of determining an area for which geo-referenced spatial data is to be saved based at least in part on data received from the at least one environmental modeling device comprises: Detecting that the geographic reference spatial data of the at least one environment modeling device is an obstacle-free area and treating it as an obstacle-free area, the method further includes: comparing a current accuracy of the determined position and orientation of the machine at the work site with the accuracy of the determined position and orientation of the machine at the work site at the time of previously saved geo-referenced spatial data; and Whether the current accuracy is almost the same, updating the stored geo-referenced spatial data in the accessible area.

11. The method according to claim 1, characterized in that: The method further comprises: parsing, by the at least one position determining unit that determines the position and orientation of the machine in the work site, data related to at least one of: a tracking device, an environmental modeling device, an object, or another machine; and The parsed data transfer is at least one of the following: a portion of data associated with a corresponding tracking device, environmental modeling device, object, or another machine; or Data capable of being received by at least one position determining unit.

12. The method according to claim 1, characterized in that The method further comprises: Initialize the tracking device in the following way: If the tracking device is disposed on at least one of the machine or another machine, determining a position and orientation of the tracking device in at least one of the machine coordinate system or the worksite coordinate system, and If the tracking device is located external to any machinery, the position and orientation of the tracking device in the worksite coordinate system is determined.

13. The method according to claim 1, characterized in that The method further comprises: Initialize the environment modeling device by If the environmental modeling device is disposed on at least one of the machine or another machine, determining a position and orientation of the environmental modeling device in at least one of the machine coordinate system or the worksite coordinate system, and If the environmental modeling device is located external to any machinery, the position and orientation of the environmental modeling device in the worksite coordinate system is determined.

14. The method according to claim 1, characterized in that The determined situational awareness is: at least one of spatial data, geo-referenced spatial data, regional work period data, regional work stage data, completion data, at least one of the position, orientation, driving direction or alternative driving direction of any machine in the work site, at least one of the position, orientation or driving direction or alternative driving direction of at least one of at least one of a stationary or moving machine, an obstacle or object to be avoided or of interest, or at least one of the surrounding environment.

15. The method according to claim 1, characterized in that The at least one tracking device also tracks the position of any other trackable marker attached to at least one of the machinery, an obstacle, or an object relative to the tracking device; and The at least one environmental modeling device also tracks the position of at least one of the following relative to the at least one environmental modeling device: at least one reference point in the work site, at least one marker point attached to the machine, any other trackable marker attached to at least one of the machine, the obstacle, or the object.

16. The method according to claim 1, characterized in that The method further comprises: determining a minimum level of accuracy of the determined position and orientation of the machine at the work site; determining a threshold level above the minimum accuracy level, and wherein If the accuracy level is below the threshold level, and if the work task being performed can be performed without moving the lower carriage, controls that move the lower carriage of the machine are disabled.

17. The method according to claim 1, characterized in that The data relating to at least one of the at least one tracking device or the at least one environmental modeling device also includes at least one of the following: data from sensors mounted on at least one of the at least one tracking device or the at least one environmental modeling device, data from sensors mounted on attachment points of the devices, data parsed by at least one of any position determination unit or any device by tracking the corresponding device or as a result of any calculation related to the corresponding device, or the accuracy level or validity of at least one of the previous.

18. The method according to claim 1, characterized in that If the at least one tracking device is arranged on the work site and if it comprises tracking means for tracking the position of the tracking device using one or more GNSS antennas, The tracking device further comprises at least one of the following as a tracking device: a camera, a stereo camera, a laser radar, a radar or a speedometer.

19. The method according to claim 1, characterized in that The determination of the position and orientation of the machine in the work site by the at least one position determination unit is additionally based at least in part on data received from one or more sensors mounted on at least one of the machine or the other machine; wherein the sensors include at least one of the following: the position, orientation, inclination, heading or travel distance of at least one of the machine or the other machine.

20. The method according to claim 1, characterized in that The machine is an excavator; and wherein The determination of the position and orientation of the machine in the work site by the at least one position determining unit is additionally based at least in part on data received from one or more sensors mounted on an upper carriage of at least one of the machine or another machine, wherein The sensor includes at least one of: a position, orientation, inclination, or heading of an upper carriage of at least one of the machine or another machine.

Images