Underground vehicle monitoring system field

Abstract

According to an example aspect of the present application, there is provided a method comprising obtaining route plan information indicative of a set of route points of a tunnel system of an underground worksite for at least partially autonomous driving of a vehicle, and the method further comprises performing, for at least some of the set of route points: detecting space information indicative of a space required by the vehicle at the associated route point; generating a set of envelopes based on the space information, wherein an envelope is indicative of a space required by the vehicle at the associated route point; and controlling visualization of the set of envelopes in a tunnel model to represent a planned route trajectory of the vehicle when driving through the route point.

Description

TECHNICAL FIELD

[0001] The present invention relates to monitoring of underground vehicles, in particular autonomous operating vehicles assigned a task to execute driving commands at a work site. BACKGROUND

[0002] A mining or construction excavation site, such as an underground hard rock or soft rock mine, can comprise areas for automated operation of mobile work machines, such as loading and / or haulage machines and drilling machines, which can also be referred to as (mining) vehicles. Such vehicles can be unmanned mining vehicles, for example remotely controlled from a control room, or manned mining vehicles, i.e. operated by an operator in a cab of the vehicle. The vehicles can be configured to autonomously execute at least some of the tasks. Automated work machines operating in an autonomous mode can operate independently without external control at least for certain parts of the work tasks or driving commands, but can be under external control in certain operating areas or operating conditions, for example during an emergency situation.

[0003] The work site and the autonomous operating vehicles at the work site can comprise a large number of mobile and stationary sensors continuously collecting data related to or affecting the operation in the mine. Such data can be referred to as mining operation data and comprise vehicle operation state data, such as speed, position at the work site, motor parameters, load, etc. and / or gallery environment data, such as temperature, air conditions, etc. The data can be transmitted to a data processing system, which can be configured to provide a mine operation control system comprising a user interface for a user of the system, which can be referred to as an operator. When needed, the operator monitoring the vehicles and manually controlling the vehicles can be instructed of the position of the vehicle executing the driving commands of the vehicle. The mine can be very large and complex and have a fleet of vehicles operating simultaneously monitored by the operator. SUMMARY

[0004] The invention is defined by the features of the independent claims. Some specific embodiments are defined in the dependent claims.

[0005] According to a first aspect of the present application, there is provided an apparatus comprising: means configured to obtain route plan information indicative of a set of route points of a tunnel system of an underground worksite for at least partially autonomous driving of a vehicle, wherein the means are further configured to, for at least some of the set of route points: detect space information indicative of a space required by the vehicle at the relevant route point; generate a set of envelopes based on the space information, wherein an envelope is indicative of a space required by the vehicle at the relevant route point; and control visualization of the set of envelopes in a tunnel model to represent a planned route trajectory of the vehicle when driving through the route points.

[0006] According to a second aspect of the present application, there is provided a method for facilitating autonomous operation vehicle monitoring and control, comprising: obtaining route plan information indicative of a set of route points of a tunnel system of an underground worksite for at least partially autonomous driving of a vehicle, the method further comprising, for at least some of the set of route points: detecting space information indicative of a space required by the vehicle at the relevant route point; generating a set of envelopes based on the space information, wherein an envelope is indicative of a space required by the vehicle at the relevant route point; and controlling visualization of the set of envelopes in a tunnel model to represent a planned route trajectory of the vehicle when driving through the route points.

[0007] Embodiments of the method include various embodiments of the apparatus of the first aspect, some of which are illustrated in dependent apparatus claims.

[0008] According to a third aspect, there is provided an apparatus comprising at least one processor, at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: obtaining route plan information indicative of a set of route points of a tunnel system of an underground worksite for at least partially autonomous driving of a vehicle, further comprising, for at least some of the set of route points: detecting space information indicative of a space required by the vehicle at the relevant route point; generating a set of envelopes based on the space information, wherein an envelope is indicative of a space required by the vehicle at the relevant route point; and controlling visualization of the set of envelopes in a tunnel model to represent a planned route trajectory of the vehicle when driving through the route points.

[0009] According to a fourth aspect, there is provided a computer program, computer program product or computer readable medium comprising computer program code configured to, when executed in data processing means, cause the apparatus to perform the method or embodiments of the method. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 Methods according to at least some embodiments are described;

[0011] Figure 2 Methods according to at least some embodiments are described;

[0012] Figure 3 Example systems according to at least some embodiments are shown; and

[0013] Figure 4 Example devices capable of supporting at least some embodiments of the invention are shown. DETAILED DESCRIPTION

[0014] A simplified example of an underground work site is shown, in this example it is an underground mine comprising a network of underground galleries. A plurality of mobile objects, e.g. personnel or pedestrians and / or mobile work machines, hereafter also referred to as vehicles, can be present in and move between different areas or operating zones of the work site.

[0015] The term "mine" herein is intended to include various underground or surface excavation sites. The vehicles can be any type of mobile work machines suitable for use in mining operations, such as trucks, dumpers, wagons, mobile rock drills or rock cutters, mobile bolting machines and shovel loaders. The vehicles can be autonomous work machines which in their autonomous mode of operation can operate / drive independently without continuous user control, but which can be brought under external control during e.g. emergency situations.

[0016] The work site comprises a communication system, e.g. a wireless access system comprising a wireless local area network (WLAN) and / or a cellular communication network (e.g. 4G, 5G or another generation cellular network), comprising a plurality of wireless access nodes 8, e.g. WLAN access points or cellular base stations. The access nodes 8 can communicate with wireless communication units comprised by the work machines or carried by the pedestrians, and with further communication devices (not shown), such as network devices configured to facilitate communication with on-site (underground or above ground) and / or remote control systems.

[0017] The system can comprise or be connected to further networks and / or data processing systems, such as a work site management system, a cloud service, data analysis devices / systems, an intermediate communication network (e.g. the Internet) etc. The system can comprise or be connected to further devices or control units, such as handheld user units, vehicle units, work site management devices / systems, remote control and / or monitoring devices / systems, data analysis devices / systems, sensor systems / devices etc.

[0018] For example, the server of the system can be configured to manage at least some operations on the worksite, for example to provide a UI for an operator to remotely monitor, and when required, control automatic operation runs of the work machines and / or to allocate work tasks for a fleet of vehicles and to update and / or monitor task execution and status. Thus, the work machines can be unmanned, the user interface can be remote from the work machines, and the work machines can be remotely monitored or controlled by an operator close to the work machines (e.g. in a tunnel) or in a control room of the worksite or even a long distance away from the worksite via a communication network.

[0019] The worksite can also include various other types of mine site operational devices, for example connectable to the control system via access nodes 8, which are not further shown. Examples of such additional mine site operational devices include various devices for power supply, ventilation, air condition analysis, safety, communication, and other automation devices. For example, the worksite can include a pass control system comprising pass control units (PCUs) separating operational zones, some of which can be set for autonomously operating work machines. The pass control system and associated PCUs can be configured to allow or prevent movement of one or more work machines and / or pedestrians between zones.

[0020] A 3D (tunnel) model of the underground worksite can be generated and stored in the control system, showing the floor, walls and roof of the tunnel. The 3D model can comprise point cloud data generated on the basis of scanning the tunnel system, or can be formed on the basis of point cloud data generated on the basis of scanning the tunnel system. The 3D model can be stored in a database accessible by one or more modules of the computing device, for example a tunnel model processing module, a user interface or visualization module, a route planning module and / or a positioning service module. In other embodiments, the 3D model can be a design model, or can be generated on the basis of a design model created by mine design software (e.g. CAD model) or on the basis of a 3D model created on the basis of tunnel lines and sections designed in drilling and blasting design software (e.g. MineSight®). Thus, the same analysis or processing can be performed on a measured or initially planned model of the tunnel system.

[0021] In complex 3D environments, such as underground mines, using a full 3D model of the mine system can be too complex and resource consuming. For example, to implement more efficient route calculation or position tracking of vehicles or pedestrians on a map comprising only the floor of the mine, possibly with attributes associated with some or all floor points. The term “floor model” generally refers to a model comprising a set of points indicating the floor of the mine tunnel at least in horizontal plane, i.e. 2D or x, y coordinates. The points can also be referred to as floor points. A 3D model of the mine tunnel can comprise point cloud data generated on the basis of scanning the mine tunnel, and the floor model is a point cloud model of the floor, comprising a subset of points extracted from the 3D point cloud data for representing the floor of the mine tunnel. As presently disclosed, the floor model can be applied as a map for mobile object motion tracking, and the floor points can thus be considered as map points. As presently disclosed, the floor model can be applied as a map for mobile object motion tracking, and the floor points can thus be considered as map points. The floor model can also comprise vertical plane, i.e. height or z coordinate data and / or supplementary data for at least some of the floor points.

[0022] Thus, a mine tunnel model applied for worksite and route plan visualization can comprise only a part of a full 3D model, such as a floor model defining floor level points and possibly also wall points. Further, the visualization can be based on a 2D model or representation of the floor system.

[0023] A drive plan or route plan or driver commands can define a route to be driven by the vehicle, and can be used as input for automatic control of the vehicle. The plan can define a start point, an end point and a set of route points for the automatic driving. A route point entry can comprise at least 2D coordinates of the route point, but can also comprise vertical coordinates or vertical layer indications. The route point entry can also comprise further information, such as speed information or obstacle and / or safety control related information. The plan can comprise information of loading areas or loading points, and can comprise data for controlling the loading of the bucket. The plan and included route point positions can be defined based on a teaching drive performed by manually driving the vehicle or by calculation based on operator input, mine tunnel model and vehicle size information. The plan can be sent to or otherwise loaded to the vehicle via a wired or wireless connection, to a memory of the vehicle for access by a control unit of the vehicle.

[0024] A vehicle, such as a vehicle, can be provided with an obstacle detection function or unit, which can be part of a collision avoidance or prevention system. The obstacle detection function can be configured to perform a collision check based on scan data received from at least a scanner configured to perform a scan of the environment of the vehicle. For example, one scanner can cover a rear portion of the vehicle and another scanner can cover a front section of the vehicle, e.g. by directional beams. The scanner can be a 3D scanner, in which case 3D scan data, such as point cloud data, is produced. The scanner can be a laser scanner or another type of sensor device suitable for determining obstacles and distances to obstacles of the vehicle, such as a 4D or another type of radar. The obstacle detection can apply to one or more obstacle detection or safety zones around the vehicle. If an object is detected as an obstacle in the zone, the vehicle can stop.

[0025] The scan results can be used to detect the position and orientation of the vehicle and one or more other elements thereof, such as a scanner or a bucket. A control unit in the vehicle can compare the scanned tunnel profile data to reference profile data stored in a tunnel model and locate the vehicle based on finding a match in the environment model and / or correct the location by dead reckoning.

[0026] The vehicle can be unmanned. Thus, the user interface can be remote from the vehicle and the vehicle can be remotely controlled by an operator in a control room in the tunnel or mine area or even at a long distance from the mine over a communication network. A control unit external to the vehicle, such as in a control system, can be configured to perform at least some of the features shown below. However, at least some of the features below can be performed on the vehicle.

[0027] It can be very challenging for an operator to monitor several simultaneously operating and driving mining vehicles in a possibly very large and complex underground work site. Several mine parts or driving situations, such as a loading operation, an unloading operation or driving through a narrow passage part, can simultaneously require the attention of the operator and the operator needs to prioritize and select which mine parts and situations make him / her to pay attention based on his / her experience and possibly manually control the vehicle. There is now provided an improvement for monitoring an autonomously operating mining vehicle tasked to perform driving commands in an underground work site.

[0028] Figure 1A method according to some embodiments is shown. The method can be performed by a mine control system or device, e.g. an apparatus of a control system, and at least one processing unit thereof. The method can be implemented by a device configured for processing route plans and generating visualizations of planned or travelled routes, e.g. a server, a worksite operator, a designer or controller workstation, a mobile unit, e.g. a mobile cellular device or other kind of mobile communication device, a vehicle on-board control device or other kind of suitably configured data processing device. The device can be configured to execute a UI generation algorithm which can execute a route and vehicle passage visualization procedure.

[0029] The method comprises obtaining 200 route plan information indicative of (locations of) a set of route points of a tunnel system of an underground worksite for at least partially autonomous driving of a vehicle. The route plan information can define at least two-dimensional coordinates of the route points. The route plan information can be obtained, e.g. by receiving the route plan information from another apparatus, route planning unit or module or memory, for monitoring during vehicle travel. The route plan information can be obtained by generating the route plan information in block 200, and thus route plan generation and subsequent visualization related features can be performed together.

[0030] Detecting 210 space information indicative of a space required by the vehicle at a relevant route point of the set of route points. This can comprise receiving the space information or calculating the (expected) space based on external vehicle dimensions and (expected) state of the vehicle at the relevant route point, some further example embodiments are shown below. The space information can be defined for at least some of the route points of the set of route points, e.g. for each route point of the set of route points. The space information can comprise (route point specific) records each identifying a route point and defining a space required by the vehicle at the given or respective route point.

[0031] For at least some of the route points of the set of route points, e.g. for each route point of the set of route points, a set of (route point specific) envelopes is generated 220 based on the space information. Thus, an envelope of the set of envelopes is indicative of a space required by the vehicle at a relevant (or respective) route point. The route point or another reference point dependent on the route point can be used as a reference for the envelope. The route point or reference point can be a center point of the vehicle (or vehicle part) expected to be at the respective location in the tunnel, around which the envelope can be generated (based on vehicle dimension data and vehicle state).

[0032] Block 230 includes controlling the visualization of the set of envelopes in the tunnel model to represent the planned route trajectory of the vehicle as it travels through the route points. This can include or involve displaying a visualization of a trajectory based on the set of envelopes and a visualization of the relevant tunnel portion based on the tunnel (or environment) model data (and mapped to the visualization of the relevant tunnel portion). Thus, the space required by the vehicle at a set of route points along the forward route can be shown to facilitate active monitoring and control of the vehicle prior to the vehicle reaching a given route point.

[0033] The planned route trajectory can generally refer to an illustration of the space required by the vehicle in the upcoming future route as the vehicle will travel through the route points. For example, the planned route trajectory can also be referred to as a trajectory along the future route (points) or a future / upcoming route (vehicle) trajectory or footprint. To indicate the planned route (or upcoming) vehicle trajectory, a graphical user interface (GUI) element can thus be generated in block 230 based on the set of envelopes. In a simple example embodiment, the set of envelopes in the set of envelopes are combined to form a substantially uniform and continuous GUI form / element to be used to represent the planned route trajectory. Thus, the size of the visualized planned route trajectory at a given route point depends on the space required by the vehicle at said given route point.

[0034] The envelope can generally refer to a 2D or 3D area that indicates the space required by the vehicle at the relevant route location. Thus, the envelope can extend from the outside dimensions of the vehicle to visualize the space expected to be required by the vehicle at the route point. The envelope GUI element can be applied to visualize a given envelope, the size of which depends on the space required by the vehicle at the route point. For a route point at which the vehicle is currently positioned, the envelope GUI element can be aligned with the vehicle GUI element or model and displayed (at least partially) around or surrounding the vehicle GUI element or model. The envelope and / or the resulting trajectory can be displayed when a real vehicle is traveling the route or when a simulated or test vehicle is traveling, e.g. when designing a route plan. However, the envelope and / or trajectory can be visualized independently of the vehicle representation or position to show the upcoming and past footprint (or trajectory) required by the vehicle at different route portions.

[0035] It should be noted that various additional information can be applied and likewise shown, for example as zones, or as additional envelopes and resulting trajectories. An example of such a zone that can be shown at least partially around the vehicle is a safety zone or obstacle detection for monitoring obstacles. The obstacle detection zone can have a minimum distance outside the envelope (from the vehicle exterior dimensions) to prevent potential false obstacle detection due to the vehicle's own structure (e.g. movement of booms, buckets, etc.). In example embodiments, the outer boundary of the envelope can be used as the minimum distance or boundary for the obstacle detection zone. The route point specific envelope is shown to facilitate the operator detecting specific parts of the route ahead that require special attention and controlling the operation of the vehicle and / or modifying the route plan information or vehicle control parameters for these specific parts and route points. The envelope can also be referred to as a route point specific sweep area or a location area affected by the vehicle at the route point.

[0036] It is noted that the envelope generation can be performed for each route point in the set of route points. The envelope generation can be repeated for all or some of the route points of the route plan, i.e. the envelope can be calculated (in advance) for the route points. Alternatively, the blocks can be repeated for the route points as needed, e.g. during driving and in accordance with the vehicle's progress, and the set of route points can even comprise a single route point. The planned route trajectory based on a selected set of route point specific envelopes ahead of the vehicle's driving direction (or even all along the route) can be displayed and updated as the vehicle progresses along the route, in accordance with the set or operator selected view option and the view being displayed.

[0037] A simplified top view display example is shown in which the vehicle is driving between tunnel walls along a route indicated by route points. The vehicle in this example is a articulated vehicle driving in a direction and comprising a front section and a rear section connected by a joint, e.g. a loader or a load haul dump (LHD) vehicle comprising a bucket (not shown separately). However, it will be appreciated that features of the present invention can be applied to monitoring various other types of vehicles.

[0038] The envelope generated based on the method of Figure 1 The envelope is shown ahead of the vehicle (towards the driving direction). The envelope can be specific to the route point. As the vehicle progresses further, the display view is continuously updated with the vehicle UI model being repositioned to a subsequent route point and the envelope being updated and defined in relation to the new route point. Likewise, obstacles can be visualized. Obstacles can be detected based on processing of the tunnel model or based on driving of an obstacle detection function monitoring an obstacle detection zone, which is visualized by indicators.

[0039] As further shown, a set of envelopes can be applied to visualize the planned route trajectory of the vehicle ahead along the upcoming route based on subsequent route points ahead of the direction of travel. It will be appreciated that although separate envelopes are shown, a substantially uniform shape defined by curves along the edges of the envelopes can be generated and displayed based on the set of envelopes, such that separate envelopes are not shown. Only a small number of envelopes are shown, but it will be appreciated that many more envelopes, and even envelopes for the complete route, can be visualized depending on the selected view.

[0040] The method can also have other modifications and implementation options, some of which will be further illustrated below. For example, a visualization of the route points in the tunnel model representing the tunnel system can also be generated. Thus, route point indicators can be mapped into the tunnel model based on the route point locations in the route plan information, e.g., as an additional block before or after block 230.

[0041] The tunnel model and spatial information of block 210 can be processed to detect at least one route point that satisfies at least one operator attention trigger condition. Thus, for at least some of the route points, the relevant tunnel model information and the space required by the vehicle at the given route point can be processed to detect specific (operator attention required) route points that exceed one or more thresholds set according to the operator attention trigger conditions.

[0042] Based on this processing, operator attention indicators can be generated for the detected at least one route point. The display of the operator attention indicators can be controlled at the relevant route points that can be visualized in the tunnel model. Various visualization or other attention-drawing methods and outputs can be used for the attention indicators, such as specific windows, flashing, coloring, etc. The attention indicators can also apply audible indicators. For example, the tunnel model processing attention triggers can be detected in conjunction with the route plan. Thus, the method can include generating / displaying the operator indicators in response to the received / detected triggers. The operator alert-related features can be additional blocks of the method after blocks 210, 220, or 230.

[0043] The trigger condition can comprise at least a distance between the vehicle and the tunnel wall at the relevant route point. The (shortest) distance between the vehicle and the tunnel wall or between the envelope and the tunnel is determined at different route points. The operator attention indicator can depend on the determined distance between the given envelope and the tunnel wall. Thus, the indicator can be generated based on the distance, e.g. selecting an indicator related to the distance range the determined distance falls into from a set of available indicator options. For example, a vehicle corner or other part of the route where the vehicle will be close to the tunnel wall or other route part can be specifically indicated to the operator and the operator can effectively focus on these parts of the route when controlling the travelling vehicle. In a simple example, a small corner (of the envelope or another information element) with a distance less than a relevant threshold value can be displayed in red.

[0044] In an example embodiment, the distance to the tunnel wall (or other obstacle) represented by the tunnel model can be calculated based on projecting a set of rays from different vertical plane positions in the tunnel model. The ray projection operation generally refers to a ray-surface intersection test. The set of rays can thus comprise tunnel wall detection rays. The tunnel wall detection rays can be projected on both sides of the vehicle model to detect the (shortest) distance to the tunnel wall on both sides of the examined route point. Based on processing the determined distances, the vehicle (model) and the route point can be located in the center between the tunnel walls at the tunnel position.

[0045] The tunnel model can comprise 3D point cloud data generated based on scanning the tunnel. In block 230, the distance to the tunnel wall (or another obstacle) in the direction of the ray projection can be determined based on a set of nearest / adjacent points. The simulation of the intersection point can be performed by measuring the distance to the adjacent point at different points of the ray, i.e. at different ray distances, e.g. every 10 cm. The threshold distance for registering a hit can be configured based on the density of the point cloud model. When at least one point, possibly multiple, is closer than the threshold distance, a hit can be registered at the ray point / distance and thus the intersection point.

[0046] The trigger condition can comprise a speed defined for the vehicle for one or more route points, and / or the operator attention indicator depends on the vehicle speed for the route point. Thus, the operator can be warned to focus on monitoring and controlling such specific points, e.g. in case the vehicle speed is substantially limited based on the route planning module and / or the safety control system. Different speed areas or ranges can be associated with different UI elements. In a simple example, a path portion of the vehicle with a low speed, e.g. within an area covered by the route trajectory based on the envelope, can be displayed in green and a path portion with a high speed exceeding a threshold speed value is displayed in red.

[0047] Thus, the operator can immediately pre-recognize where special attention should be paid in the planned route trajectory, and whether a corrective action is needed. For example, the operator can adjust the speed at a given route point or route section. Because the visualization can be updated immediately after the operator input (and e.g. a corrective action regarding route point position or speed), the operator can immediately get the information and provide further control input if the corrective action is sufficient.

[0048] The presently disclosed features help the operator to more efficiently monitor and control a plurality of simultaneously driving vehicles in a mine area, which can be very complex and extensive. The operator can be provided with a GUI view to facilitate fast detection of major problem or hazardous sections along the route (vehicles with major problem or hazardous sections) from less problematic sections and prioritize actions. The features of the invention can also improve the help to the operator to detect existing or expected bottlenecks on the planned route and provide immediate reaction or proactive action control input. Furthermore, the production efficiency can be improved when the space can be more efficiently utilized and the route further optimized for autonomous operating vehicles (e.g. because larger vehicles can be used, driven faster or the required size of the gallery can be reduced).

[0049] The space required by the vehicle detected in block 210 can refer to determining the space based on a set of input parameters, or receiving such space information from another entity calculating the space, e.g. a route plan or a controller module of the vehicle. The set of points can be applied as route sections and stored and processed as a spline, which can reduce the processing requirements. Thus, Figure 2 The features illustrated in Fig. 3 (including envelope generation and subsequent envelope set visualization) can be based on and applied to the set of points (e.g. a spline). Thus, the term route point can refer to not only a single geographical point, but also to a route section defined by a set of geographical points.

[0050] For the envelope at a relevant route point, the space required by the vehicle can be determined based on processing vehicle dimension data and data regarding the expected state of the vehicle at the relevant route point. For example, the dimension data can be specific to a vehicle class / type, model, or specific to each vehicle. It should be noted that a dynamic or predetermined margin can be added on the outer dimensions of the vehicle. Furthermore, the envelope can be specified in various forms. For example, a drill rig can have an envelope extending from a rectangular form covering the area of one or more drilling booms. The expected state can include the vehicle speed at the route point, and if the environment sensing is related to vehicle coordinates, also the steering angle a of the vehicle at the route point. However, if the gallery wall information is related to work site coordinates, also other vehicle state parameters related to defining the envelope can be included, e.g. vehicle position, heading and orientation.

[0051] In block 220, two or even more envelopes or sub-envelopes can be generated for each relevant route point. Thus, for each route point, the set of envelopes can comprise two or more (sub-)envelopes. In block 230, these envelopes can be applied to visualize the planned route trajectory, or one or more further envelopes are displayed as supplemental information (layer).

[0052] In an example embodiment, with reference to the above examples, for a articulated vehicle, at a relevant route point, a front section envelope can be generated based on the space required by the front section, and a rear section envelope can be generated based on the space required by the rear section. The envelopes can be generated based on the dimensions of the respective machine sections and dynamic information, such as speed at the respective route point and / or articulation angle between the front section and the rear section. The display of the front section envelope and the rear section envelope is controlled to be visually separated, such as by different colors or other visual differentiation methods. The envelopes can be displayed at different vertical plane positions, i.e. with different vertical (z) direction coordinates.

[0053] An example is shown in which only a route point based trajectory path is shown in the tunnel model 80. It can be seen that it can be particularly difficult for inexperienced operators to detect potential problematic sections in a complex underground tunnel system.

[0054] An example is shown in which a planned route front (body) section trajectory based on a set of front section envelopes and a rear (body) section trajectory based on a set of rear section envelopes are generated and displayed. The envelopes and / or trajectories can be displayed as visually separated and / or at different vertical plane levels. It has been noted that these provide substantial help, in particular for inexperienced operators to understand the machine space requirements and how close the machine will be to obstacles in narrow tunnel sections, and facilitate effective and focused monitoring and control of the vehicle fleet.

[0055] The tunnel model can be a 3D model, and the envelopes are displayed as 2D or 3D layers on the route points (which can extend to cover a set of route points or a spline, as already noted). As already shown, multiple envelopes can be displayed on a route point, as well as further auxiliary information. The envelopes and potentially other information can be displayed as specific separate layers on the route points. As the amount of information on a route point increases, effective visualization and separation of different information becomes crucial. To more effectively detect different information, multiple layers at different vertical plane positions (at different vertical / z direction positions) are applied.

[0056] Figure 2 A method is shown that can be applied in combination with Figure 1 a method as inFigure 1 a further step of the method of Figure 2 The blocks of show operations for a single waypoint, but it is understood that these operations can be repeated for at least some of the set of waypoints to visualize additional waypoint-specific information in addition to or as part of the planned route trajectory visualized in block 230.

[0057] In addition to the first envelope (e.g. based on which the planned route trajectory can be generated), a second envelope is generated 500 for a given waypoint. The second envelope can depend on (and be based on) an expected or defined state of the vehicle at said waypoint, e.g. the speed of the vehicle. The layer order or positioning parameters can be configured as control parameters that influence the positioning of the layers in the display view.

[0058] A vertical plane order and position can be determined 510 for the waypoint, the first envelope and the second envelope. The selectively prioritized layer can be dynamically positioned on top. Thus, the method can further comprise:

[0059] - determining a first vertical plane position of the first envelope at the relevant waypoint, wherein the first vertical plane position is different from a second vertical plane position defined for the waypoint in the visualized tunnel model. Preferably, the first and second vertical plane positions are also different from a vertical plane tunnel floor level or portion defined by the tunnel model at the waypoint,

[0060] - determining at least one third vertical plane position of the second envelope, and

[0061] - controlling the display of the first envelope at the first vertical plane position, the waypoint at the second vertical plane position and the second envelope at the third vertical plane position at the relevant waypoint.

[0062] In one example of prioritization and ordering of the information layers (from bottom to top in the vertical plane z-direction), the waypoint or waypoint spline can be located above the tunnel floor level, the first envelope above the waypoint, and the second envelope above the first envelope.

[0063] These operations can be repeated for successive waypoints and envelopes, and the resulting trajectory on different vertical plane (z) levels is obtained. It is understood that more than two layers can be generated and positioned by applying the above described method. Further, instead of an envelope, block 500 can comprise generating a vehicle state indicator or operator attention indicator, which is then positioned at a different vertical plane position than the first envelope. Another example comprises:

[0064] - generating an obstacle detection zone indicator indicative of an obstacle detection zone determined for the vehicle for monitoring the presence of an obstacle at a route point of the set of route points, wherein the obstacle detection zone depends on an expected state of the vehicle at the route point,

[0065] - determining a third vertical plane position of the obstacle detection zone indicator, and

[0066] - controlling the display of the obstacle detection zone indicator at the third vertical plane position at the route point.

[0067] Further, it is not necessary to display the route point, but in some UI generation embodiments only the envelope and potentially additional route point specific information is displayed at the route point position in the tunnel model.

[0068] An example 3D visualization is shown in which at least two layers or planned route trajectories at different heights are shown, i.e. with different (z) vertical plane height levels (for simplicity only that (thin) part of the planned future trajectory). These trajectories can be generated based on the set of envelopes (e.g. the front body envelope and the rear body envelope). However, one of the layers can comprise and be generated on the basis of non-envelope based input. For example, the layer can show an obstacle detection zone or an operator attention indicator. In a further example, a selector can be provided for the operator by which the trajectory or layer to be displayed can be selected.

[0069] In yet another example, the different information layers / envelopes are ordered and displayed as follows (from bottom to top in the vertical plane z direction):

[0070] - displaying the road (at least the tunnel pit bottom surface) and potentially additional 3D environment defined in the tunnel model as it shows the real environment displaying known obstacles and tunnel walls,

[0071] - displaying the rear body envelope and the resulting planned route trajectory slightly above the road surface (which can be positioned slightly above the typical road surface shape). Thus, the surface of the planned envelope visualization can be kept consistent and the map / environment visualization does not stick to it.

[0072] - displaying the front body envelope and the resulting planned route trajectory slightly above the rear body envelope so that the operator sees it better. If they are different, the difference to the rear body peak can be positioned lower than it.

[0073] - displaying the route spline above the front body envelope, preferably, thus it is clearly shown on top.

[0074] - display a collision or obstacle detection zone indicator above the route spline, as it is generally considered to be the most important for real-time situations and allows the operator to quickly detect the reason for e.g. the vehicle deciding to stop.

[0075] It should be appreciated that the above order is just one example, and various other orders of the layers and envelopes can be applied, and one or more of them can be omitted (or additional layers / envelopes applied). It should be noted that the envelopes can be used as input for automatic control of the vehicle. These envelopes can be used as input for collision prevention functionality and obstacle detection. Another example is that the steepness of the envelope can be used as input for a slope decelerator that automatically decelerates the vehicle when e.g. driving down a tunnel slope.

[0076] The operator can be provided with options to provide input to obtain additional information, change the path or other route plan parameters at one or more of the relevant route points, control the path of the vehicle and / or control the vehicle at or before the route points.

[0077] In addition to visualizing the planned route trajectory and the route parts that require the attention of the operator, there are various additional actions that can be invoked to help the operator when designing or testing a route plan or monitoring an autonomously operating vehicle in a worksite. Some additional examples include providing guidance for the operator, generating a corrective control action for the vehicle or suggesting a control action or a route (point) parameter change for the operator.

[0078] The input can be provided to a data processing unit (DPU) via a display or another input device and a GUI interface, which is configured to perform at least GUI related processing in dependence of the user input. The GUI processing can be performed by a GUI processing module, which is configured to generate or at least control the GUI that is displayed by the display to the operator via the GUI interface. The GUI processing module can be configured to perform at least some of the above described features, such as blocks 220-230 and 500-520.

[0079] The control system, such as the DPU, can be configured to detect the position of the vehicle, e.g. based on position data from the vehicle or a positioning service. This can be performed for some or all vehicles at the worksite. The position of the vehicle is mapped into the tunnel model. A vehicle model can be displayed and the envelope visualization is made in the tunnel model based on the mapped position.

[0080] The DPU or associated control system can also include a vehicle control module or unit configured to generate control commands to the vehicle on-board control system based on associated user input after displaying 230, 420 the envelope and other route point specific information for a particular vehicle. In response to receiving (user) control input from the operator via the input device, the control commands are transmitted to the vehicle to control at the event of a particular route point or autonomous task, for example to overcome an alert or underperformance issue.

[0081] After executing the control commands in the relevant vehicle, new vehicle and / or driving command related data (e.g. positioning information) can be received by the DPU and at least some of the above described features can be repeated, e.g. blocks 220 and 230. An updated vehicle operation status view can then be displayed to the operator, which can include an updated envelope and a generated planned route trajectory visualization. The earlier displayed operator attention indicator can also be updated according to the updated received data and can even be removed if the operator attention triggering condition no longer exists.

[0082] The vehicle status related data can be processed to detect at least one corrective control action for the vehicle and / or route plan to address the detected situation and condition that caused the operator attention. This can include defining control actions for one or more vehicles, for example. Control information for mapping vehicle status condition or event cause information to one or more operator guidance elements and / or corrective actions can be stored in the data storage of the DPU. Control signals and / or content of the operator guidance elements can thus be generated or selected based on the control information. For example, selecting a control command, guidance information record or data element or further event characterization information that matches the vehicle type and alert identifier.

[0083] In response to detecting that the necessary conditions for automatic control are met, a control signal associated with the determined control command and mine operation device can be transmitted. Alternatively, the operator can be instructed of the corrective action and associated vehicle and control command, for example by generating an operator guidance GUI element. In embodiments, the operator is provided with an input option via which the operator can directly trigger transmission of the determined control signal.

[0084] It will be appreciated that various further features can supplement or distinguish at least some of the above described embodiments. For example, there can be further user interaction and / or automation functionality to further assist the operator in designing a route plan or monitoring and controlling the vehicles and operation / settings of the vehicles.

[0085] Figure 3An operations module of a mine operations control device or system (e.g., server 81) is shown in accordance with some embodiments. The object tracking module 83 can be configured to track the location of moving objects and provide 3D position indicators to other modules, such as the location services module 82.

[0086] The server 81 can include a task manager or management module 84 configured to manage at least some operations at the work site. For example, the task manager can be configured to assign work tasks to a fleet of work machines and update control signals, send the control signals to the work machines, and / or monitor work machine task performance and status, which is indicated in the task management GUI.

[0087] The server 81 can include a model processing module 85 that can maintain one or more models of the underground work site (e.g., a 3D tunnel model). The model processing module 85 is configured to map vehicle models and associated envelopes to the tunnel model.

[0088] The server 81 can include a GUI module 86 configured to generate at least some display views for operators (local and / or remote). The GUI module 86 can be configured to generate 3D (and / or 2D) views based on the 3D model or tunnel floor model by applying at least some of the above-illustrated embodiments, which include current locations of vehicles, associated envelope visualizations, and operator attention indicators.

[0089] The server 81 can include additional modules 88, such as remote monitoring processes and UIs, event processing modules configured to process mine operations data, and / or cloud scheduler components configured to provide selected work site information (e.g., vehicle monitoring information) to cloud services.

[0090] The system and server 81 can be connected to additional systems 90 and / or networks 89, such as work site management systems, cloud services, intermediate communication networks (e.g., the Internet, etc.). The system can also include or be connected to additional devices or control units, such as handheld user units, vehicle units, work site management devices / systems, remote control and / or monitoring devices / systems, data analysis devices / systems, sensor systems / devices, etc.

[0091] The object tracking 83 can be implemented as part of another module (e.g., the location services module 82). The location services 82 is configured to provide moving object location information obtained from or generated based on information from the object tracking 83 to related other modules or functions (e.g., the database 87, the visualization graphical user interface 86, and / or remote units or systems 70) via one or more networks 89, either upon request or by push transmission. In Figure 3In the examples of Figs. 1-3, the modules are shown interconnected, but it should be understood that not all modules need to be connectable.

[0092] The system can comprise or be connected to a control unit or module of the working machine or another mine operating device, e.g. can transmit control commands to the control unit or module of the working machine or another mine operating device. In example embodiments, a control unit can be provided in each autonomously operating vehicle and configured to control at least some autonomous operations of the vehicle based on received control commands.

[0093] The electronic device comprising electronic circuitry can be an apparatus for implementing at least some embodiments of the application, e.g. in connection with the method shown in Figure 1 The apparatus can be comprised in at least one computing device connected to or integrated into a worksite control or automation system or vehicle. The apparatus can be a distributed system comprising a set of at least two connectable computing devices. In connection with the method shown in Figure 1 (And / or Figure 1 At least one of the features shown in connection with (and / or embodiments of) the method can be performed in a first device and other features can be performed in a second device, which are connected via wireless and / or wired connection. At least some of the features can be performed in a server or other type of control unit available to an operator, which remotely controls the vehicle and / or generates route point data for the vehicle. For example, the envelope generation (blocks 200-220) can be performed in a first device, such as a server or a safety control device, and the visualization and display of the envelope can be performed in a second device, such as a vehicle or a UI control device.

[0094] In some example embodiments, edge computing is applied, whereby some features (e.g. blocks 200-210 / 220) can be performed at an edge node, which can be located e.g. at the vehicle. For example, the processing of the 3D scan data can be performed at the edge node. The edge node can perform the localization function and update and / or generate (in case of SLAM) the tunnel model. The edge node can perform the route generation, which can also generate the envelope along the route. The edge node can perform the collision prevention related features, including the obstacle detection. The obstacle detection function (by the edge node or another control unit) can receive as input the envelope, the scan data and the machine dynamic limits and detect whether the vehicle can collide with an object. The edge node (or another control unit) in the vehicle can control the real-time communication of the vehicle position and state data to a controller unit, in which the monitoring function can be configured to slow down or stop the machine when needed.

[0095] Figure 4An example device capable of supporting at least some embodiments of the application is shown. A device 100 is shown, which can be configured to perform at least some of the embodiments related to the features described above in relation to vehicle monitoring and envelope display, e.g. Figure 2 The device 100 can comprise or implement a DPU, for example.

[0096] A processor 91 is comprised in the device 100, which can comprise a single- core or multi-core processor, for example. The processor 91 can comprise more than one processor. The processor can comprise at least one application-specific integrated circuit, ASIC. The processor can comprise at least one field-programmable gate array, FPGA. The processor can be configured, at least partly, by computer instructions executed in the processor, to perform actions.

[0097] The device 100 can comprise a memory 92. The memory can comprise random access memory and / or persistent memory. The memory can be at least partly accessible by the processor 91. The memory can be at least partly comprised in the processor 91. The memory can be at least partly external to the device 100, but accessible by the device. The memory 92 can be a means for storing information, e.g. parameters 94 affecting the operation of the device. The parameter information can comprise, among others, parameter information affecting the display elements and envelope generation and / or visualization, e.g. threshold values. The memory 92 or another memory or storage means connectable to the device 100 can also comprise input data to be processed by the device, e.g. route plan files, vehicle dimension data and / or tunnel models as applied above.

[0098] The memory 92 can comprise computer program code 93 comprising computer instructions configured to be executed by the processor 91. When computer instructions configured to cause the processor to perform certain actions are stored in the memory and the device is generally configured to run under the direction of the processor using the computer instructions from the memory, the processor and / or at least one processing core thereof can be considered configured to perform said certain actions. The processor can form, together with the memory and the computer program code, a means for performing at least some of the above-mentioned method blocks in the device.

[0099] The apparatus 100 can comprise a communication unit 95 comprising a transmitter and / or a receiver. The transmitter and receiver can be configured to transmit and receive, respectively, i.e. in particular vehicle monitoring related data and control commands, in accordance with at least one cellular or non-cellular standard. The transmitter and / or receiver can be configured to operate in accordance with, for example, Global System for Mobile Communications, GSM, Wideband Code Division Multiple Access, WCDMA, Long Term Evolution, LTE, 3GPP New Radio Access Technology (N-RAT), Wireless Local Area Network, WLAN, and / or Ethernet. The apparatus 100 can comprise a Near Field Communication, NFC, transceiver. The NFC transceiver can support at least one NFC technology, such as NFC, Bluetooth or similar technologies.

[0100] The apparatus 100 can comprise or be connected to a UI. The UI can comprise at least one of a display 96, a loudspeaker, an input device 97, such as a keyboard, a joystick, a touch screen, and / or a microphone. The UI can be configured to display a view based on the worksite model and the mobile object position indicators. A user can operate the apparatus and control at least some features of the control system, e.g. the illustrated system. Figure 4 The user can control the vehicles 4-7 and / or the server through the UI, e.g. change operating modes, change display views, modify parameters 94 in response to user authentication and appropriate permissions associated with the user, etc.

[0101] The apparatus 100 can further comprise and / or be connected to additional units, devices and systems, such as one or more sensor devices 98 sensing the environment of the apparatus 90.

[0102] The processor 91, the memory 92, the communication unit 95 and the UI can be interconnected by electrical leads inside the apparatus 100 in a number of different ways. For example, each of the above-mentioned devices can be individually connected to a main bus inside the apparatus to allow the devices to exchange information. However, as will be appreciated by a person skilled in the art, this is merely one example and various ways of interconnecting at least two of the above-mentioned devices can be chosen according to embodiments without departing from the scope of the application.

[0103] It should be understood that the disclosed embodiments of the present application are not limited to the particular structures, process steps, or materials disclosed herein but are extended to equivalents thereof The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0104] Reference throughout this specification to one embodiment or an embodiment means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases in

[0105] As used herein, a plurality of items, structural elements, compositional elements, and / or materials can be presented in a common list for convenience. However, these lists should be construed as though each member of the list is individually identified as a separate and unique member. Thus, no individual member of such list should be construed as a de facto equivalent of any other member of the same list solely based on their presentation in a common group without indications to the contrary. In addition, various embodiments and examples of the present application can be referred to herein in terms of a specific placement of one or more parts of a structure or composition. As should be understood by those of ordinary skill in the art, such references can refer to the placement of those parts or one or more parts at any level or position in or on a structure or composition. The application encompasses any permutation or combination of one or more parts, or one or more features of one or more embodiments.

[0106] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details were provided, such as examples of lengths, widths, shapes, etc., to provide a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the application.

[0107] While the forgoing examples are illustrative of the principles of the application in one or more particular applications, it will be apparent to those of ordinary skill in the art that numerous modifications can be made without departing from the principles of the application and the generic scope of the application. Other modifications, equivalents, and alternatives, within the spirit and scope of the application, will be apparent to those skilled in the art from this disclosure and descriptions thereof. Therefore, it is intended that the claims not be limited to the described embodiments but rather encompass all such modifications, equivalents, and alternatives.

[0108] The verbs “comprise” and “include” are used as open-ended limitations in this document, both in reference to the claims and describing the application. The member recited following the terms is not meant to be the only member or to exclude other, additional members. Unless otherwise indicated, the features, structures, or characteristics described in connection with one embodiment can be combined with those of another embodiment. Furthermore, unless otherwise indicated, the description of a variety of embodiments can be made without loss of generality. One skilled in the art will recognize that the applications can be practiced with one or more of the described embodiments, structures, components, variations, alternatives, and / or features. Thus, the present applications are not intended to be limited to the embodiments described herein but rather are to be accorded the widest scope consistent with the principles and numerous embodiments disclosed herein.

Claims

1. An apparatus comprising means including at least one processor and at least one memory, the at least one memory including computer program code, the at least one memory and the computer program code being configured to execute the apparatus using the at least one processor, the apparatus being configured to perform the following operations: - Obtain route planning information that defines a route to be traveled by an articulated vehicle, the articulated vehicle comprising a front portion and a rear portion articulated to the front portion, the route planning information indicating a set of route points in a tunnel system of an underground construction site for at least partial autonomous driving of the articulated vehicle. - Detect spatial information of at least some of the route points in the set of route points, the spatial information indicating the space required by the front and rear portions of the articulated vehicle at the relevant route points based on the dimensions of the corresponding machine parts and the hinge angle between the front and rear portions. - Generate a set of route point-specific envelopes based on the spatial information, wherein the front partial envelope indicates the space required by the front portion at the relevant route point, the rear partial envelope indicates the space required by the rear portion at the relevant route point, and - The visualization of the specific envelope of the set of route points in the control tunnel model at the relevant route points to represent the front and rear portions of the planned route of the articulated vehicle as it travels through the route points.

2. The device according to claim 1, wherein, The device is also configured to perform: - Process the tunnel model and the spatial information to detect at least one route point that satisfies at least one operator attention trigger condition. - Generate an operator attention indicator for the detected at least one route point, and - For the detected at least one route point, control the display of the operator attention indicator in the tunnel model.

3. The device according to claim 2, wherein, The at least one triggering condition includes at least a distance between the articulated vehicle and the tunnel wall or other obstacle at a route point in the set of route points, and / or the operator attention indicator depends on a determined distance between the envelope of the specific envelope of the set of route points and the tunnel wall or other obstacle.

4. The device according to claim 2 or 3, wherein, The at least one triggering condition includes the speed of the articulated vehicle at at least one of the set of route points, and / or the operator attention indicator depends on the speed defined for the articulated vehicle at the at least one route point.

5. The device according to claim 3, wherein, Detecting the space includes: based on processing vehicle size data and data about the expected state of the articulated vehicle at the relevant route points, determining the space required by the articulated vehicle at the relevant route points for an envelope in a specific envelope of the set of route points.

6. The device according to claim 5, wherein, The expected state includes the vehicle speed and steering angle at the relevant route points.

7. The device according to any one of claims 1-3, wherein, The device is also configured to: - Determine a first vertical plane position for an envelope within the set of route point-specific envelopes, wherein the first vertical plane position differs from a second vertical plane position, the second vertical plane position being defined for visualizing additional envelopes, vehicle status indicators, or route points associated with the envelope within the set of route point-specific envelopes, and - Control the display of the specific envelope of the set of route points at the first vertical plane position in the tunnel model and the additional envelope at the second vertical plane position, the vehicle status indicator, or the route points.

8. The device according to any one of claims 1-3, wherein, The device is also configured to: - Generate an obstacle detection zone indicator, which indicates an obstacle detection area determined by the articulated vehicle for monitoring the presence of obstacles at route points in a set of route points, wherein the obstacle detection area depends on the expected state of the articulated vehicle at the route points. - Determine the position of the third vertical plane for the obstacle detection zone indicator, and - Control the display of the obstacle detection zone indicator at the third vertical plane position at the route point.

9. The device according to any one of claims 1-3, wherein, The device is also configured to visually separate the display control of the front partial envelope and the rear partial envelope.

10. The device according to any one of claims 1-3, wherein, The tunnel model is a three-dimensional model indicating the tunnel system, and the set of route point-specific envelopes are combined and displayed as two-dimensional or three-dimensional layers located on the route points according to layer order or positioning parameters.

11. An underground construction site monitoring system, comprising one or more data processing devices, one or more user interface devices, one or more data storage devices, and one or more communication devices, wherein the system includes the equipment according to any one of claims 1 to 10.

12. A method comprising: - Obtain route planning information that defines a route to be traveled by an articulated vehicle, the articulated vehicle comprising a front portion and a rear portion articulated to the front portion, the route planning information indicating a set of route points in a tunnel system of an underground construction site for at least partial autonomous driving of the articulated vehicle. - Detect spatial information of at least some of the route points in the set of route points, the spatial information indicating the space required by the front and rear portions of the articulated vehicle at the relevant route points based on the dimensions of the corresponding machine parts and the hinge angle between the front and rear portions. - Generate a set of route point-specific envelopes based on the spatial information, wherein the front partial envelope indicates the space required by the front portion at the relevant route point, the rear partial envelope indicates the space required by the rear portion at the relevant route point, and - Visualize the specific envelope of the set of route points in the control tunnel model to represent the front and rear portions of the planned route of the articulated vehicle as it travels through the route points.

13. The method of claim 12, further comprising: - Process the tunnel model and the spatial information to detect at least one route point that satisfies at least one operator attention trigger condition. - Generate an operator attention indicator for the detected at least one route point, and - For the detected at least one route point, control the display of the operator attention indicator in the tunnel model.

14. The method according to claim 12 or 13, further comprising: Based on the processing of vehicle size data and data about the expected state of the articulated vehicle at the relevant route points, the space required by the articulated vehicle at the relevant route points is determined for the envelope in the specific envelope of the set of route points.

15. A computer-readable medium comprising computer program code for causing the device to perform the method of any one of claims 12 to 14 when executed in a data processing apparatus.

Images