Vision system and method for vehicles

Abstract

Disclosed is a system for processing image information configured to be connected to a vehicle, particularly a combat vehicle The system comprises a plurality of image sensing modules configured for sensing an outside image data set. The plurality of image sensing modules comprises a high-resolution sensor module, at least one or a plurality of first-type image sensing module(s) and at least one second-type image sensing module. The system further comprises a data-processing system configured for receiving the outside image data set from the image sensing modules, and processing the outside image data set. Further disclosed is a vehicle which comprises the system.

Description

Vision system and method for vehiclesField[1] The present invention relates to the field of the vision systems for vehicles, particularly for heavy vehicles, vehicles for offroad use or combat vehicles.Background[2] Armored vehicles, as well as some heavy vehicles and offroad vehicles offer only limited visibility for a crew and / or passengers. This is for example due to an armored hull resulting in extremely small openings or the need of periscopes, a high chassis providing sufficient ground clearance but in exchange reducing visibility of the direct environment of the vehicle, as well as the vehicle superstructure or accessory equipment. In other words, such vehicles often compromise the field of vision (FoV) of the driver for other features relating to other functions of the vehicle, such as protection of crew and passengers or provision of functional modules.[3] In consequence, a situational awareness of the crew and / or the passengers may be limited. Poor situational awareness may affect every all soldier in mechanized and armored units. Better situational awareness may reduce threats and enable all troops to make timely, good decisions to fulfil their mission more effectively. Improved situational awareness improves the force to pass the OODA (observation, orientation, decision, action) loop more rapidly, thus decreasing reaction times and allowing for appropriate reactions in developing situations. With increased SA and fusion of information the observation and orientation phases will be passed much faster and that leads to better and timely decisions that will end up in more effective action. Situational awareness may is created on all levels by all members of the an armored force. Further, situational awareness may be relevant for planning, collection, analysis and dissemination of information. Hence, in case of military vehicles, increased situational aware may increase overall performance of the vehicle as well as other parts of the force. Likewise, poor situational awareness may increase a threat to the troops next to or at the front line.[4] Similarly, in case of vehicles required to manoeuvre particularly offroad or in difficult or dangerous terrain, the field of vision of the crew may be essential for passing obstacles. In particular in case of vehicles configured for navigating in narrow spaces like urban environments, the FoV may be more relevant. This particularly holds true for armored vehicles configured for operation in such environments, such as counterinsurgency armored vehicles.[5] EP 3182051 Al discloses a method for displaying images on a head-mounted display (HMD) device that compensates for a user's vestibulo-ocular reflex (VOR) response. The displayed HMD image is compensated for predicted eye position such that the displayed image stays centered on the fovea of the eye, during transient eye movement caused by head motion, resulting in better display readability, discernment and cognitive processing.[6] WO 2023281488 Al discloses a transfer-alignment system for a Head-Mounted Display (HMD), and a display coupled to the HMD, wherein the display is adapted to display images rendered by a display-processor, and wherein the HMD is monitored by a tracking system configured to provide information indicating position and / or orientation of the HMD with respect to a Frame of Reference (FoR), the system comprising: at least one first inertial sensor attached to the HMD and configured to acquire HMD's Inertial Readings Information (IRI); at least one second inertial sensor attached to the display and configured to acquire display's IRI; and a processor configured to: obtain the HMD's IRI, the display's IRI and the information indicating the HMD's position and / or orientation with respect to the FoR; continuously analyse movement information of the HMD and the display to determine relative orientation therebetween; and cause the display-processor to adjust the images to conform with respect to the FoR.[7] EP 3019968 Al discloses a system, a method, a computer program and a combat vehicle with an integrated system, for processing of tactical information. The system comprises at least one sensor for registration of at least one image sequence display at least a portion of the surroundings of the combat vehicle. The system further comprises a navigation module arranged to register a current position of the vehicle. The system further comprises a tactical data module for storage of tactical information and an information processing unit arranged to process said image sequence to superimpose the tactical information onto said image sequence.[8] EP 3865809 Al an apparatus to improve a situational awareness of a pilot or driver controlling a vehicle by means of a control appliance (200). The control appliance (200) comprises a display (210) for depicting surroundings of the vehicle, and the vehicle comprises a missile warner (300) and a sensor for fine tracking (FTS) (410), wherein the FTS (410) is configured to provide high-resolution images for a tracking of an approaching missile detected by the missile warner (300). The apparatus comprises a control unit (100), configured to couple a directable line of sight of the FTS (410) with the display (210), and to employ the high-resolution images of the FTS (410) to improve the depiction of the surroundings of the vehicle on the display.[9] While the prior art approaches may be satisfactory in some regards, they have certain shortcomings and disadvantages.

[0010] One obvious way to increase the field of vision would be a use of transparent armor, e.g., bulletproof glass. While this may be feasible in some cases, bulletproof glass fails to provide protection above protection level 3, while being a costly solution. Also, survivability against several hits is hard to achieve. Thus, providing an unobstructed direct visibility of the surroundings for the crew and / or the passengers is technically cumbersome, if not impossible.

[0011] Another problem in combat vehicles or vehicles configured for resolving different tasks in addition to transport is an availability of data from different sensors, which may lead to a data overload of the vehicle crew, who may in consequence fail to optimally use the available data. For example, in the case of combat vehicles, modern battlefield is also characterized by data overload from different sensors, such as cameras, UAV feeds, voice comms, video downlinks and the like. In today's systems, this data is usually presented through different interfaces and channels, such as intercom, periscopes, windows, monitors, paper documents etc. To the applicant's knowledge, there is functional solution available on the market that overcomes the above challenge.

[0012] Further, onboard offroad vehicles and particularly armored vehicles, usually, space is quite limited, which makes use of big screens impractical. At the same time, screens often fail to be sufficiently durable and rugged to withstand shocks from movement in rough terrain or from indirect or even direct fire.Summary

[0013] It is therefore an object of the invention to overcome or at least alleviate the shortcomings and disadvantages of the prior art. More particularly, it is an object of the present invention to provide a system for improved situational awareness as well as a vehicle comprising an improved situational awareness.

[0014] In a first embodiment, a system for processing image information is disclosed. The system is configured to be connected to a vehicle. The system comprises a plurality of image sensing modules configured for sensing an outside image data set. The system further comprises a data-processing system configured for receiving the outside image data set from the image sensing modules, and processing the outside image data set.

[0015] The data-processing system may for example be a mission computer.

[0016] The vehicle may for example be a combat vehicle, such as an infantry fighting vehicle, an armored personal carrier, a combat support platform, an engineering vehicle or a main battle tank.

[0017] Sensing the outside image data set may refer to sensing data and generating an outside image data set based thereon. The outside image data set may comprise video data. The outside image data set may also comprise image data comprising single images.

[0018] The plurality of image sensing modules may comprise a high-resolution sensor module. The high-resolution sensor module may be connected to the data-processing system by means of a CoaXpress-connection.

[0019] The high-resolution sensor module may be a driver sensor system, i.e., a sensor system oriented in a forward direction of the vehicle.

[0020] The high-resolution sensor module may comprise at least two infrared sensors, particularly at least three infrared sensors. The infrared sensors may be configured for generating thermal image data.

[0021] In the context of this disclosure, the term "infrared sensor" is intended to refer to sensors suitable to sense electro-magnetic radiation in the infrared spectrum which is suitable to determine a temperature of an object in the range of -20° to +100°C.

[0022] The term "thermal image data" may relate to image data generated by infrared sensors. In other words, the thermal image data may be image data obtained by thermal imaging.

[0023] Infrared sensors may optionally advantageously provide for night vision as well as for identifying heat signatures. Also, infrared radiation may be suitable to provide vision through smoke, dust and certain fogs.

[0024] The infrared sensors may be long-wavelength infrared sensors. That is, the infrared sensors may be configured for sensing radiation comprising a wavelength of 8pm to 15pm.

[0025] The infrared sensors, particularly the long-wavelength infrared sensors, may be uncooled.

[0026] The high-resolution sensor module may comprise at least two image sensors, particularly at least three image sensors. The image sensors may be configured for generating image data based on at least one of visible light and near-infrared radiation. The image sensors may be electro-optical image sensors.

[0027] The term "near-infrared radiation" is intended to refer to electro-magnetic radiation comprising a wavelength of 0.75 pm - 1.4 pm.

[0028] In other words, the term "image sensor" may refer to sensors configured for sensing electro-magnetic radiation lying within the spectrum comprising a wavelength of 0.4 pm - 1.4 pm throughout the present disclosure, e.g., a sensor configured for sensing visible lightor a sensor configured for sensing near-infrared radiation. The term "image data" may refer throughout the present disclosure to image data generated by image sensors configured for sensing electro-magnetic radiation comprising a wavelength lying within the spectrum of 0.4 pm - 1.4 pm.

[0029] Combining at least one image sensor and at least one infrared sensor may optionally advantageously allow to combine an extended sensing range of the image sensor with a means to detect heat radiation and heat signatures in the surroundings.

[0030] The image outside data set may be based on or comprise the image data and the thermal image data.

[0031] The infrared sensors of the high-resolution sensor module may be arranged to cover a horizontal field of view of at least 165°, particularly at least 180°. The image sensors of the high-resolution sensor module may be arranged to cover a horizontal field of view of at least 165°, particularly at least 180°.

[0032] When, in the present disclosure, reference is made to the field of view, without an indication of "horizontal" or "vertical", the horizontal field of view, that is, an angle of a horizontal plane covered by the field of view, is meant, unless the context clearly dictates another meaning.

[0033] Thus, optionally advantageously, the high-resolution sensor module may provide for improved supervision of a forward direction of the vehicle and surroundings in the forward direction, i.e., the direction in which the vehicle typically moves with a highest velocity and in which a driver needs to see sufficiently far to make decisions in time.

[0034] The field of view of the infrared sensors and the image sensors of the primage image sensing module may be substantially identical. In other words, a substantially identical angular area in a horizontal plane may be covered with respect.

[0035] The high-resolution sensor module may be configured to be mounted to a front of the vehicle.

[0036] The image sensors of the high-resolution sensor module may each comprise a resolution of at least 1920 x 1080 px, particularly at least 2500 x 1400 px. The indication "px" is intended to refer to pixels. Thus, 1920 x 1080 px is intended to specify a resolution of 1920 pixels by 1080 pixels. The disclosed resolution may optionally advantageously provide for improved visibility at long ranges and / or for improved visibility of details, such as writing or signs.

[0037] The infrared sensors of the high-resolution sensor module may comprise a resolution of at least 640 px x 480 px. Thus, a good tradeoff between a data load as well as cost andnecessary resolution can be provided, as long wave infrared sensors are in most cases not suitable to detect very small or remote details.

[0038] The plurality of image sensing modules may further comprise at least one or a plurality of first-type image sensing module(s). Particularly, the plurality of image sensing module may comprise at least two first-type image sensing modules.

[0039] The first-type image sensing module(s) may also be referred to as close range sensor module, daylight sensor module or electro-optical sensor module. The first type image sensing module may for example be oriented in side directions of the vehicle, such as towards the left and the right.

[0040] The first-type image sensing module(s) is not intended to specify that the system must necessarily comprise said first-type image sensing module(s). Rather, their presence is optional.

[0041] The first-type image sensing module(s) may be connected to the data-processing system via ethernet.

[0042] Each first-type image sensing module may comprise an image sensor configured for generating image data based on visible light. With respect to the terms "image sensor" and "image data" as well as with respect to advantages of these features, reference is made to the above discussion in the context of high-resolution sensor module.

[0043] In other words, additional image data may be generated by the image sensor of the first-type image sensing module.

[0044] The image sensor of the first-type image sensing module may for example be a fullHD-camera.

[0045] Each first-type image sensing module may comprise an image sensor configured for generating image data based on visible light. The image sensors of the first-type image sensing module may comprise a resolution of at least 1024 px x 768 px, preferably at least 1280 px x 720 px.

[0046] A sensor module comprising such a resolution may optionally advantageously provide good visibility in close range despite a limited data throughput. Further, such a sensor module may be optionally advantageous for data transmission to remote locations for the same reason.

[0047] In some embodiments, the image sensors of the first-type image sensing modules may optionally not comprise a resolution of more than 1920 px x 1080 px.

[0048] The first-type image sensing module(s) may be configured to be mounted to at least one, preferably two sides of the vehicle. In other words, the system may comprise a first- type image sensing module mounted to a left side of the vehicle, and a first-type image sensing module mounted to a right side of the vehicle.

[0049] Each of the first-type image sensing modules may cover a horizontal field of view of at least 165°, particularly at least 175°.

[0050] Each first-type image sensing module may comprise a single image sensor configured for generating image data based on visible light.

[0051] The first-type image sensing module(s) may be configured for observing a close proximity of the vehicle.

[0052] The plurality of image sensing modules may further comprise at least one second- type image sensing module.

[0053] The second-type image sensing module may also be referred to as dual close range sensor system. The second-type image sensing module may be configured for monitoring the close proximity of the vehicle, e.g., on a rear side. However, for example in setups with bandwidth restrictions, the second-type image sensing module may also be used as a driver sensor system, i.e., used to monitor a space before the vehicle. The person skilled in the art will easily understand that in some embodiments, the system may comprise at least one second-type image sensing module, but no first type-image sensing module or vice versa, and that the terms "first-type" and "second-type" are not intended to transfer a preference or dependence between these two types of sensor modules.

[0054] The second-type image sensing module may comprise an infrared sensor. The infrared sensor may be configured for generating thermal image data. In other words, additional thermal image data may be generated by the infrared sensor of the second-type image sensing module.

[0055] The infrared sensor of the second-type image sensing module may be a long- wavelength infrared sensor.

[0056] Further, the second-type image sensing module may comprise an image sensor. The image sensor may be configured for generating image data based on at least one of visible light and near-infrared radiation. In other words, additional image data may be generated by the image sensor of the second-type image sensing module.

[0057] With respect to the terms "image data", "image sensor", "thermal image data" and "infrared sensor" as well as with respect to advantages of these features, reference is made to the discussion in the context of the high-resolution sensor module.

[0058] Using the second-type image sensing module on a rear side of the vehicle in addition to the high-resolution sensor module may optionally advantageously provide for the advantages of this combination in all directions of drive, while saving bandwidth without compromising situational awareness, as the vehicle typically does not move as fast in a rear direction as in a forward direction.

[0059] The infrared sensor of the second-type image sensing module may be arranged to cover a horizontal field of view of at least 60°, particularly of at least 80°, such as about 86°. The image sensor of the second-type image sensing module may be arranged to cover a horizontal field of view of at least 120°, particularly at least 125°, such as about 130°.

[0060] The second-type image sensing module may be configured to be mounted to at least one of a rear or the front of the vehicle. However, the second-type image sensing module or a plurality thereof may also be mounted to a side of the vehicle, in particular in setups where thermal vision is required around the vehicle. This may be optionally advantageous for vehicle variants modified for night combat, night-time operations or operations with particularly bad sight, e.g., firefighting.

[0061] The image sensor of the at least one second-type image sensing module may comprise a resolution of at least 1024 x 768 px, preferably at least 1280 x 720 px.

[0062] The infrared sensors of the at least one second-type image sensing module may comprise a resolution of at least 640 x 480 px.

[0063] The image sensing modules may be configured to be placed on the vehicle to cover a horizontal field of view of at least 270°, preferably 315°, still more preferably 360° around the vehicle. The indicated angle may relate to a horizontal angle around the vehicle within which objects around the vehicle can be detected or sensed by at least one of the image sensing modules at, e.g., a distance of 5 m from the vehicle.

[0064] Each image sensing module may comprise a housing. In other words, the high- resolution sensor module, the first-type image sensing module(s) and the at least one second-type image sensing module may each comprise a housing.

[0065] Each of the image sensing modules may comprise a cleaning component.

[0066] Each image sensing module may comprise a connection to a water provision system. Each cleaning component of the image sensing modules may comprise a waterjet cleaning component for cleaning the respective module's sensor(s). The vehicle or the system may comprise the water provision system including a pump and corresponding conduits.

[0067] Each image sensing module may comprise a connection to a pressurized air system. Each cleaning component of the image sensing modules may comprise a pressurized aircleaning component for cleaning the respective module's sensor(s). The vehicle or the system may comprise the pressurized air system. The pressurized air system may be an onboard pressurized air system of the vehicle.

[0068] Each cleaning component and / or each pressurized air cleaning component may be integrated into the housing of the respective image sensing module.

[0069] The high-resolution sensor module may comprise at least one wiper.

[0070] The second-type image sensing module may comprise at least one wiper.

[0071] The at least one wiper may be integrated or mounted to the housing of the respective sensing module / sensor module.

[0072] Each image sensing module may comprise a heating. The heating may be user- controlled. Thus, optionally advantageously, frost may be removed and vision may be ensured under corresponding atmospheric conditions.

[0073] Each of the image sensing modules may comprise a mechanical vibration dampening.

[0074] Each housing may comprise water and dust protection at least according to IP68, particularly IP69.

[0075] In line with EN 60529, IP68 may correspond to a protection against ingress of dust and survival of immersion in water at a depth of at least 1 meter. In line with EN 60529, IP69 may correspond to a protection against ingress of dust and survival of powerful high- temperature water jets, e.g., due to cleaning of the vehicle.

[0076] The system may comprise at least one or a plurality of head-mounted display(s) (HMDs). The head-mounted display(s) may be worn at a user's head. The head-mounted display(s) may also be helmet-mounted display(s).

[0077] The HMD(s) may be VR-glass(es).

[0078] The HMD(s) may be configured for receiving image data, particularly the processed image data, from the data-processing system.

[0079] The at least one or the plurality of HMD(s) may be at least three HMDs, particularly four HMDs. Thus, situational awareness of the whole vehicle crew, typically comprising 3- 4 crew members, may be improved.

[0080] The system may be configured for determining a relative orientation, particularly a relative pose, of the HMD(s) with respect to the vehicle. In other words, the system may be configured for determining an orientation of the HMD(s), particularly a pose of the HMD(s), with respect to a coordinate system that is fixed with respect to the vehicle.

[0081] The HMD(s) may be configured for 6-degrees-of-freedom eye- and / or headtracking. In other words, the HMD(s) may be configured for determining the pose of a user's head, containing three rotational and three translational coordinates.

[0082] The HMD(s) may be configured for focus adjustment of -4D to +4D, that is, -4 dioptres to +4 dioptres. This may optionally advantageously allow for adjustment to an individual sight of the user(s).

[0083] Dioptres may comprise the unit 1 / m and correspond to the definition in directive 71 / 354 / EEC in the version of 2020-06-13; Annex I, sec. 4.

[0084] Each HMD may comprise a horizontal field of view of at least 30°, particularly at least 35°, such as 37°. In other words, the displayed output may be visible in a field of vision of at least 30°, particularly 35°, such as 37°, of the user.

[0085] Each HMD may comprise an output resolution of at least 1200 px x 800 px, preferably at least 1280 px x 960 px.

[0086] The system may be configured to be integrated into and / or mounted to the vehicle. In particular, the vehicle may be a combat vehicle. Further, the vehicle may be configured for night missions.

[0087] The data-processing system may comprise an on-board data-processing system. In some embodiments, the data-processing system may be the on-board data-processing system.

[0088] For example, the on-board data-processing system may be configured for executing a 3D-engine, such as the Unity (TM) engine, OpenGL-based engines, an Ogre3D engine or another custom-built 3D-engine. In particular, the 3D-engine may be configured for multiview rendering of input video data.

[0089] The on-board data-processing system may be configured for multi-threading and intentional load splitting between one or more CPUs and one or more GPUs.

[0090] The data-processing system may be configured for processing a relative orientation, particularly a relative pose, of the image sensing modules with respect to the vehicle.

[0091] The relative orientation of the image sensing modules with respect to the vehicle may comprise a pitch angle, a yaw angle and a roll angle with respect to a coordinate system of the vehicle.

[0092] Processing said relative position, particularly said relative pose, may for example comprise at least one of storing or loading a preset, receiving calibration datacorresponding to the relative orientation and / or relative pose, and calibrating the relative orientation and / or relative pose.

[0093] The data-processing system, particularly the on-board data-processing system, may be configured for generating processed image data for each of the HMD(s) based on the processed outside image data set.

[0094] The image sensing modules may be configured to be mounted to an outside of the vehicle. The high-resolution sensor module may be oriented substantially in a front direction of the vehicle. At least three image sensing modules may be oriented in other directions of the vehicle, such as a rear direction, a left and a right direction.

[0095] The person skilled in the art will easily understand that the three image sensing modules may be selected from first-type and second type image sensing modules, e.g., only first-type image sensing modules, only second-type image sensing modules, or a quantity of first-type and second-type image sensing modules.

[0096] In other words, e.g., the vehicle may comprise one or at least one high-resolution sensor module and at least three image sensing modules selected from first-type and second-type image sensing module(s), e.g., two first-type image sensing modules on the left and right side and one second-type image sensing module on the rear side, optionally advantageously providing for day and night vision in all driving directions and for long range vision in the main driving direction. This setup may be further optionally advantageous in case of direct control of the vehicle, i.e., control of the vehicle by a crew in the vehicle.

[0097] The system is further configured for determining a relative orientation of each HMD with respect to the vehicle, particularly a relative pose of each HMD with respect to the vehicle. The data-processing system may further be configured for generating the processed image data for each HMD by deriving image data from the outside image data set corresponding to a substantially unobstructed view of an environment of the vehicle corresponding to the relative orientation, particularly relative pose, of the HMD.

[0098] In other words, the system may provide for simultaneous multi-user viewpoints of the environment of the vehicle, optionally advantageously improving situational awareness for each crew member.

[0099] For example, the data-processing system may be configured for deriving the image data for each HMD from the output image data set by means of the 3D-engine.

[0100] Determining the relative orientation of the HMD may comprise sensing an orientation of the HMD, sensing an orientation of the vehicle, and determining the relative orientation of the HMD with respect to the vehicle based thereon. However, determiningthe relative orientation of the HMD may also be based on image sensors of the HMD capturing an environment of the HMD, which environment, in use within the vehicle, moves with the vehicle.

[0101] Deriving the processed image data from the outside image data set for each HMD may comprise generating processed image data corresponding to a section of a virtual dome corresponding to the relative orientation, particularly the relative pose, of the respective HMD.

[0102] The virtual dome may be a virtual hemisphere around the vehicle corresponding to the visible environment. In other words, the section of the virtual dome may be a virtual view corresponding to the relative orientation of the HMD, particularly the relative pose of the HMD.

[0103] In a most simple implementation, this may for example be achieved by the data- processing system being configured for using a virtual projection of the outside image data set on the virtual dome. In such an implementation, the data-processing system may be configured for deriving processed image data from a section the virtual dome corresponding to the relative orientation, particularly the relative pose, of the respective HMD. However, most 3D-engines do either directly transform the input image data set or only render sections of the virtual dome that are to be outputted.

[0104] The data-processing system, particularly the on-board data-processing system, may be configured for multi-view rendering.

[0105] The data-processing system, particularly the on-board data-processing system, may be configured for using the 3D engine for deriving the processed image data.

[0106] A latency between an image being sensed by the image sensing modules and an output of corresponding processed image data by the HMD(s) may amount to at most 80 ms, particularly at most 70 ms.

[0107] A latency between an image being sensed by the high-resolution sensor module and an output of corresponding processed image data by the HMD(s) may amount to at most 50 ms.

[0108] At least one or a plurality of the HMD(s) may be vehicle HMD(s). The vehicle HMD(s) may by HMD(s) associated with the vehicle. The vehicle HMD(s) may be configured for use inside and / or in proximity of the vehicle. In particular, the vehicle HMD(s) may be configured for receiving data from the on-board data-processing system.

[0109] In other words, the HMD(s) associated with the vehicle may be referred to as vehicle HMD(s). The vehicle HMD(s) may for example be physically associated with the vehicle bymeans of a wired connection. The vehicle HMD(s) may also be associated with the vehicle by means of a direct wireless connection, such as a connection by means of a wireless LAN. In other words, the vehicle HMD(s) may be configured for receiving data from the on-board data-processing system by means of a direct connection.

[0110] The vehicle HMD(s) may be connected to the on-board data-processing system by means of a wired connection, such as a wired connection comprising an HDMI-link and / or a serial link. The serial link may for example be configured for receiving user instructions, orientation data and / or pose data. The wired connection may for example comprise a power feed.

[0111] At least one of processing the outside image data set and generating the processed image data for each of the HMD(s) may comprise fusing the thermal image data and the image data.

[0112] Fusing the thermal image data and the image data already when processing the outside image data set may be more efficient if there is the same setting for all users, i.e., if all users get an output based on the fused thermal image data and image data. Fusing the thermal image data and the image data when generating the processed image for each of the HMD(s) may be more efficient if different settings may be used for the user(s) of the vehicle, e.g., the crew members.

[0113] The data-processing system may be configured for superimposing at least one image section of the thermal image data and at least one corresponding image section of image data. In other words, the image section of the thermal image data and the at least one corresponding image section of the image data may be combined as semi-transparent layers.

[0114] The data-processing system may be configured for correcting for optical distortions in at least one of the thermal image data and the image data.

[0115] For example, distortions in the thermal image data may be corrected, distortions in the image data may be corrected, both may be corrected. In an alternative example, distortions of the thermal image data may be corrected to match the distortion in the image data or vice versa.

[0116] Generating the processed image data for each of the HMD(s) based on the outside image data set may comprise providing at least one image section relating to a rear perspective and / or a side perspective.

[0117] The rear perspective and / or the side perspectives may be with respect to the vehicle. The rear perspective and / or the side perspective may also be with respect to an orientation of the HMD.

[0118] The system may be configured for enlarging the rear perspective and / or the side perspective shown by an HMD upon receiving an instruction of a user wearing the HMD.

[0119] The data-processing system may comprise an on-board data-processing system and a remote data-processing system. This may for example be the case when the vehicle is configured for remote control and / or remote supervision.

[0120] The on-board data-processing system may be configured to be mounted to or integrated into the vehicle. The remote data-processing system may be located remote from the vehicle. The on-board data-processing may be configured for transmission of at least a part of the image data set to the remote data-processing system.

[0121] The remote data-processing system may for example be located in a mobile or stationary control center or in a data center. In some embodiments, the remote data- processing system may be a part of a cloud system.

[0122] The on-board data-processing system may be configured for generating the outside image data set. The remote data-processing system may be configured for generating processed image data for each of the HMD(s) based at least on the part of the outside image data set.

[0123] At least one or a plurality of the HMD(s) may be remote HMD(s). The remote HMD(s) may be configured for receiving data from the remote data-processing system. Particularly, the remote HMD(s) are configured for use off the vehicle, e.g., in the mobile or stationary control center.

[0124] The system may further be configured for determining an absolute orientation of the remote HMD(s), particularly an absolute pose of the remote HMD(s). The terms "absolute orientation" and "absolute pose" may refer to a pose and orientation relative to the remote location, e.g., the command center.

[0125] The data-processing system, particularly the remote data-processing system, may be configured for generating the processed image data for the remote HMD(s) by deriving image data from at least the part of the outside image data corresponding to the substantially unobstructed view of the environment of the vehicle based on the absolute orientation, particularly based on the absolute pose, of the remote HMD(s). The person skilled in the art will easily understand that deriving the image data corresponding to the substantially unobstructed view of the environment of the vehicle based on the absolute orientation, particularly based on the absolute pose, may comprise adding or subtracting an offset from said absolute orientation or absolute pose.

[0126] The image sensing modules may be configured to be mounted to an outside of the vehicle, wherein at least three, preferably at least four image sensing modules are mounted to the vehicle.

[0127] In other words, for example if the vehicle is configured for remote control or remote supervision, there may be at least four image sensing modules selected from the first-type image sensing modules, the second-type image sensing modules and first-type image sensing modules as well as second-type image sensing modules.

[0128] The at least three image sensing modules may comprise a second-type image sensing module oriented in the forward direction of the vehicle. Thus, optionally advantageously, combined image data and thermal image data may be available in a main direction of driving of the vehicle.

[0129] The on-board data-processing system may be configured for processing the outside image data set for transmission with an average bandwidth of at most 5 Mbit / s to the remote data-processing system.

[0130] This may be optionally advantageous in cases where the vehicle is configured for remote control or remote supervision in order to avoid excessing an available transmission bandwidth, e.g., transmission bandwidth from the vehicle to the command center.

[0131] Processing the outside image data set for transmission may comprise preprocessing the outside image data set, compressing the outside image data set, reducing a resolution of the outside image data set and / or cropping images contained in the outside image data set.

[0132] Generating the processed image data for each of the HMD(s) based on the outside image data set may comprise providing at least one image section relating to a rear perspective and / or a side perspective.

[0133] The system may be configured for enlarging the rear perspective and / or the side perspective shown by an HMD upon receiving an instruction of a user wearing the HMD.

[0134] The system's HMD can be configured to display a screen, showing the image data, preferably a stitched version of the environment into the direction of the user's head or any pre-selected direction.

[0135] The HMD can be configured to display an information bar fixed in position with respect to the screen and preferably located below the screen. The HMD can be further configured to display a plurality of windows arranged outside the screen and preferably above the screen.

[0136] The HMD can be further configured to display a floating window that represents the direction of view of a user.

[0137] The screen can comprise an interaction zone, preferably at each opposite side ends thereof. The interaction zone can be configured to be the zone that triggers the scrolling of the view of the screen from the side of the interaction zone towards the center of the screen when the eyetracking and / or headtracking indicates a view of the user onto the interaction zone.

[0138] The HMD can be configured to display at least two, preferably three, windows with information of interest, such as image data and / or other information.

[0139] The windows can be arranged generally above the screen and / or centered with respect to the floating window. One of the windows can be enlarged according to the activation of a user. This activation can be caused by the eyetracking and / or headtracking of a user and / or respective tracking devices tracking eyes and / or head position and / or movement.

[0140] The HMD can be further configured to display an activation zone that triggers the enlargement of the respective one of the windows when a user's eye is tracked by the eyetracking to look onto the activation zone.

[0141] The activation zone comprises the respective window and extends beyond the lower end of the respective window and preferably has a surface that is at least 1.2 larger than the surface of the window.

[0142] The enlargement of a window can at least be an enlargement with a factor of at least 1.2, preferably at least 1.5, more preferably at least 2, even more preferably at least 3.

[0143] In a second embodiment, a vehicle is disclosed. The vehicle comprises the system according to any of the system embodiments. The vehicle may be a motor vehicle, i.e., a motor-propelled vehicle.

[0144] For example, the vehicle may be a land vehicle.

[0145] The vehicle may be at least one of an offroad vehicle, a heavy vehicle and an armored vehicle.

[0146] The vehicle may be a combat vehicle.

[0147] The vehicle may be configured for data transmission to the control center, such as the control center located remotely from the vehicle.

[0148] In such embodiments, for example, the on-board data-processing system may be configured for data transmission to the remote data-processing system by means of a data-transmission component of the vehicle. The vehicle may comprise the data- transmission component.

[0149] The vehicle may be configured for being controlled from the control center.

[0150] The vehicle may be retrofitted with the system. In other words, the system may be mounted to the system after the first commissioning.

[0151] The vehicle may be configured for night missions. In other words, the vehicle may be configured for operation during nighttime and / or in darkness.

[0152] The following embodiments also form part of the invention.System embodiments

[0153] Below, embodiments of a system will be discussed. The system embodiments are abbreviated by the letter "S" followed by a number. Whenever reference is herein made to the "system embodiments", these embodiments are meant.51. A system for processing image information configured to be connected to a vehicle, particularly a combat vehicle, the system comprising a plurality of image sensing modules configured for sensing an outside image data set,- a data-processing system configured for o receiving the outside image data set from the image sensing modules, and o processing the outside image data set.52. The system according to any of the preceding embodiments, wherein the plurality of image sensing modules comprises a / at least one high-resolution sensor module.53. The system according to the preceding embodiment, wherein the high-resolution sensor module comprises at least two infrared sensors, particularly at least three infrared sensors, which infrared sensors are configured for generating thermal image data.54. The system according to the preceding embodiment, wherein the infrared sensors are long-wavelength infrared sensors.55. The system according to any of the embodiments with the features of S2, wherein the high-resolution sensor module comprises at least two image sensors, particularly at least three image sensors, which image sensors are configured for generating image data based on at least one of visible light and near-infrared radiation.56. The system according to any of the preceding embodiments with the features of S3 and / or S5, wherein the infrared sensors of the high-resolution sensor module are arranged to cover a horizontal field of view of at least 165°, particularly at least 180°, and wherein the image sensors of the high-resolution sensor module are arranged to cover a horizontal field of view of at least 165°, particularly at least 180°.57. The system according to the preceding embodiment, wherein the field of view of the infrared sensors and the image sensors of the primage image sensing module is substantially identical.58. The system according to any of the preceding embodiments with the features of S2, wherein the high-resolution sensor module is configured to be mounted to a front of the vehicle.59. The system according to any of the preceding embodiments with the features of S5, wherein the image sensors of the high-resolution sensor module each comprise a resolution of at least 1920 x 1080 px, particularly at least 2500 x 1400 px.510. The system according to any of the preceding embodiments with the features of S3, wherein the infrared sensors of the high-resolution sensor module comprise a resolution of at least 640 x 480 px.511. The system according to any of the preceding embodiments, wherein the plurality of image sensing modules further comprises at least one or a plurality of first-type image sensing module(s), particularly at least two first-type image sensing modules.512. The system according to the preceding embodiment, wherein each first-type image sensing module comprises an image sensor configured for generating image data based on visible light.513. The system according to the preceding embodiment, wherein each first-type image sensing module comprises an image sensor configured for generating image data based on visible light, wherein the image sensor of each first-type image sensing module comprises a resolution of at least 1024 x 768 px, preferably at least 1280 x 720 px.514. The system according to any of the preceding embodiments with the features of Sil, wherein the first-type image sensing module(s) are configured to be mounted to at least one, preferably two sides of the vehicle.515. The system according to any of the preceding embodiments with the features of Sil, wherein each of the first-type image sensing modules covers a horizontal field of view of at least 165°, particularly at least 175°.516. The system according to any of the preceding embodiments with the features of S12, wherein each first-type image sensing module comprises a single image sensor configured for generating image data based on visible light.517. The system according to any of the preceding embodiments with the features of Sil, wherein the first-type image sensing module(s) are configured for observing a close proximity of the vehicle.518. The system according to any of the preceding embodiments, wherein the plurality of image sensing modules further comprises at least one second-type image sensing module.519. The system according to the preceding embodiment, wherein the second-type image sensing module comprises an infrared sensor, which infrared sensor is configured for generating thermal image data.520. The system according to the preceding embodiment, wherein the infrared sensor of the second-type image sensing module is a long-wavelength infrared sensor.521. The system according to any of the embodiments with the features of S18, wherein the second-type image sensing module comprises an image sensor, which image sensor is configured for generating image data based on at least one of visible light and near-infrared radiation.522. The system according to any of the preceding embodiments with the features of S19 and / or S21, wherein the infrared sensor of the second-type image sensing module is arranged to cover a horizontal field of view of at least 60°, particularly of at least 80°, such as about 86°, and wherein the image sensor of the second-type image sensing module is arranged to cover a horizontal field of view of at least 120°, particularly at least 125°, such as about 130°.523. The system according to any of the preceding embodiments with the features of S18, wherein the second-type image sensing module is configured to be mounted to at least one of a rear or the front of the vehicle.524. The system according to any of the preceding embodiments with the features of S5, wherein the image sensor of the at least one second-type image sensing module comprises a resolution of at least 1024 x 768 px, preferably at least 1280 x 720 px.525. The system according to any of the preceding embodiments with the features of S3, wherein the infrared sensor of the at least one second-type image sensing module comprises a resolution of at least 640 x 480 px.526. The system according to any of the preceding embodiments, wherein the image sensing modules are configured to be placed on the vehicle to cover a horizontal field of view of at least 270°, preferably 315°, still more preferably 360° around the vehicle.527. The system according to any of the preceding embodiments, wherein each image sensing module comprises a housing.528. The system according to any of the preceding embodiments, wherein each of the image sensing modules comprises a cleaning component.529. The system according to the preceding embodiment, wherein each image sensing module comprises a connection to a water provision system and wherein each cleaning component of the image sensing modules comprises a water jet cleaning component for cleaning the module's sensor(s).530. The system according to any of the two preceding embodiments, wherein each image sensing module comprises a connection to a pressurized air system and wherein each cleaning component of the image sensing modules comprises a pressurized air cleaning component for cleaning the module's sensor(s).531. The system according to any of the two preceding embodiments, wherein each cleaning component and / or each pressurized air cleaning component is integrated into the housing of the respective image sensing module.532. The system according to any of the preceding embodiments with the features of S2, wherein the high-resolution sensor module comprises at least one wiper.533. The system according to any of the preceding embodiments with the features of S18, wherein the second-type image sensing module comprises at least one wiper.534. The system according to any of the preceding embodiments, wherein each image sensing module comprises a heating, particularly wherein the heating is user-controlled.535. The system according to any of the preceding embodiments, wherein each of the image sensing modules comprises a mechanical vibration dampening.536. The system according to any of the preceding embodiments with the features of S27, wherein each housing comprises water and dust protection at least according to IP68, particularly IP69.537. The system according to any of the preceding embodiments, wherein the system comprises at least one or a plurality of head-mounted display(s) (HMDs).538. The system according to the preceding embodiment, wherein the HMD(s) are VR- glass(es).539. The system according to any of the preceding embodiments with the features of S37, wherein the HMD(s) are configured for receiving image data, particularly processed image data, from the data-processing system.540. The system according to any of the preceding embodiments with the features of S37, wherein the at least one or the plurality of HMD(s) are at least three HMDs, particularly four HMDs.541. The system according to any of the preceding embodiments with the features of S37, wherein the system is configured for determining a relative orientation, particularly a relative pose, of the HMD(s) with respect to the vehicle.542. The system according to the preceding embodiment, wherein the HMD(s) are configured for 6-degrees-of-freedom eyetracking and / or headtracking.543. The system according to any of the preceding embodiments with the features of S37, wherein the HMD(s) are configured for focus adjustment of -4D to +4D.544. The system according to any of the preceding embodiments with the features of S37, wherein each HMD comprises a horizontal field of view of at least 30°, particularly at least 35°, such as 37°.545. The system according to any of the preceding embodiments with the features of S37, wherein each HMD comprises an output resolution of at least 1200 px x 800 px, preferably at least 1280 px x 960 px.546. The system according to any of the preceding embodiments, wherein the system is configured to be integrated into and / or mounted to the vehicle, particularly the combat vehicle.547. The system according to any of the preceding embodiments, wherein the data- processing system comprises an on-board data-processing system.548. The system according to any of the preceding embodiments, wherein the data- processing system is configured for processing a relative orientation, particularly a relative pose, of the image sensing modules with respect to the vehicle.549. The system according to any of the preceding embodiments, wherein the data- processing system, particularly the on-board data-processing system, is configured for generating processed image data for each of the HMD(s) based on the processed outside image data set.550. The system according to any of the preceding embodiments with the features of S2 and S46, and particularly with the features of Sil and / or S18, wherein the image sensing modules are configured to be mounted to an outside of the vehicle, wherein the high- resolution sensor module is oriented substantially in a front direction of the vehicle, and wherein at least three image sensing modules are oriented in other directions of the vehicle, such as a rear direction, a left and a right direction.551. The system according to any of the preceding embodiments with the features of S37, wherein the system is further configured for determining a relative orientation of each HMD with respect to the vehicle, particularly a relative pose of each HMD with respect to the vehicle, and wherein the data-processing system is configured for generating the processed image data for each HMD by deriving image data from the outside image data set corresponding to a substantially unobstructed view of an environment of the vehicle corresponding to the relative orientation, particularly relative pose, of the HMD.552. The system according to the preceding embodiment, wherein deriving the processed image data from the outside image data set for each HMD comprises generating processed image data corresponding to a section of virtual dome corresponding to the relative orientation, particularly the relative pose, of the respective HMD.553. The system according to any of the preceding embodiments with the features of S49, wherein the data-processing system, particularly the on-board data-processing system, is configured for multi-view rendering.554. The system according to any of the preceding embodiments with the features of S49, wherein the data-processing system, particularly the on-board data-processing system, is configured for using a 3D engine for deriving the processed image data.555. The system according to any of the preceding embodiments with the features of S49, wherein a latency between an image being sensed by the image sensing modules and an output of corresponding processed image data by the HMD(s) amounts to at most 80 ms, particularly at most 70 ms.556. The system according to any of the preceding embodiments with the features of S49 and S2, wherein a latency between an image being sensed by the high-resolution sensor module and an output of corresponding processed image data by the HMD(s) amounts to at most 50 ms.557. The system according to any of the preceding embodiments with the features of S47 and S37, wherein at least one or a plurality of the HMD(s) are vehicle HMD(s), wherein the vehicle HMD(s) are configured for use inside and / or in proximity of the vehicle, particularly wherein the vehicle HMD(s) are configured for receiving data from the on-board data- processing system.558. The system according to the preceding embodiment, wherein the vehicle HMD(s) are connected to the on-board data-processing system by means of a wired connection, such as a wired connection comprising an HDMI-link and / or a serial link.S59. The system according to any of the preceding embodiments with the features of at least one of S3 and S19, wherein at least one of processing the outside image data set andgenerating processed the image data for each of the HMD(s) comprises fusing the thermal image data and the image data.560. The system according to the preceding embodiment, wherein the data-processing system is configured for superimposing at least one image section of the thermal image data and at least one corresponding image section of image data.561. The system according to any of the two preceding embodiments, wherein the data- processing system is configured for correcting for optical distortions in at least one of the thermal image data and the image data.562. The system according to any of the preceding embodiments with the features of S49, wherein generating the processed image data for each of the HMD(s) based on the outside image data set comprises providing at least one image section relating to a rear perspective and / or a side perspective.563. The system according to the preceding embodiment, wherein the system is configured for enlarging the rear perspective and / or the side perspective shown by an HMD upon receiving an instruction of a user wearing the HMD.564. The system according to any of the preceding embodiments, wherein the data- processing system comprises an on-board data-processing system and a remote data- processing system, wherein the on-board data-processing system is configured to be mounted to or integrated into the vehicle, wherein the remote data-processing system is located remote from the vehicle, and wherein the on-board data-processing is configured for transmission of at least a part of the image data set to the remote data-processing system.565. The system according to the preceding embodiment, wherein the on-board data- processing system is configured for generating the outside image data set, and wherein the remote data-processing system is configured for generating processed image data for each of the HMD(s) based at least on the part of the outside image data set.566. The system according to any of the preceding embodiments with the features of S64 and S37, wherein at least one or a plurality of the HMD(s) are remote HMD(s), wherein the remote HMD(s) are configured for receiving data from the remote data-processing system, particularly wherein the remote HMD(s) are configured for use off the vehicle.567. The system according to the preceding embodiment, wherein the system is further configured for determining an absolute orientation of the remote HMD(s), particularly an absolute pose of the remote HMD(s).568. The system according to the preceding embodiment, wherein the data-processing system, particularly the remote data-processing system, is configured for generating the processed image data for the remote HMD(s) by deriving image data from at least the part of the outside image data corresponding to the substantially unobstructed view of an environment of the vehicle based on the absolute orientation, particularly based on the absolute pose, of the remote HMD(s).569. The system according to any of the preceding embodiments with the features of S64 and at least one of Sil and S18, wherein the image sensing modules are configured to be mounted to an outside of the vehicle, wherein at least three, preferably at least four image sensing modules are mounted to the vehicle570. The system according to the preceding embodiment, wherein the at least three image sensing modules comprise a second-type image sensing module oriented in the forward direction of the vehicle.571. The system according to any of the preceding embodiments, wherein the on-board data-processing system is configured for processing the outside image data set for transmission with an average bandwidth of at most 5 Mbit / s to the remote data-processing system.572. The system according to any of the preceding embodiments with the features of S64, particularly with the features of S68, wherein generating the processed image data for each of the HMD(s) based on the outside image data set comprises providing at least one image section relating to a rear perspective and / or a side perspective.573. The system according to the preceding embodiment, wherein the system is configured for enlarging the rear perspective and / or the side perspective shown by an HMD upon receiving an instruction of a user wearing the HMD.574. The system according to any of the preceding embodiments wherein the HMD is configured to display a screen (61) showing the image data.575. The system according to the preceding embodiment wherein the HMD is configured to display an information bar (F) fixed in position with respect to the screen and preferably located below the screen.576. The system according to any of the preceding embodiments wherein the HMD is configured to display a plurality of windows (63) arranged outside the screen (61) and preferably above the screen (61).577. The system according to any of the preceding embodiments wherein the HMD is configured to display a floating window (70) that represents the direction of view of a user.578. The system according to any of the preceding embodiments, preferably the four preceding embodiments, wherein the screen (61) has an interaction zone (61A), preferably at each opposite side ends thereof.579. The system according to the preceding embodiment wherein the interaction zone (61A) is configured to be the zone that triggers the scrolling of the view of the screen (60) from the side of the interaction zone (61A) towards the center of the screen (61) when the eyetracking and / or headtracking indicates a view of the user onto the interaction zone (61A).580. The system according to any of the preceding embodiments wherein the HMD is configured to display at least two, preferably three, windows (63) with information of interest, such as image data and / or other information.581. The system according to the preceding embodiment wherein the windows (63) are arranged generally above the screen (61) and / or centered with respect to the floating window (70).582. The system according to any of the two preceding embodiments wherein one of the windows can be enlarged according to the activation of a user.583. The system according to the preceding embodiment wherein the activation can be caused by the eyetracking and / or headtracking of a user.584. The system according to the preceding embodiment wherein the HMD is configured to display an activation zone (63D) that triggers the enlargement of the respective one (63E) of the windows (63) when a user's eye is tracked by the eyetracking to look onto the activation zone (63D).585. The system according to any of the two preceding embodiments wherein the activation zone (63D) comprises the respective window and extends beyond the lower end of the respective window and preferably has a surface that is at least 1.2 larger than the surface of the window.586. The system according to any of the four preceding embodiments wherein the enlargement of a window (63) is at least an enlargement with a factor of at least 1.2, preferably at least 1.5, more preferably at least 2, even more preferably at least 3.Vehicle embodiments

[0154] Below, embodiments of a system will be discussed. The system embodiments are abbreviated by the letter "V" followed by a number. Whenever reference is herein made to the "vehicle embodiments", these embodiments are meant.VI. A vehicle, comprising the system according to any of the system embodiments.V2. The vehicle according to the preceding embodiment, wherein the vehicle is at least one of an offroad vehicle, a heavy vehicle and an armored vehicle.V3. The vehicle according to any of the preceding vehicle embodiments, wherein the vehicle is a combat vehicle.V4. The vehicle according to any of the preceding vehicle embodiments, wherein the vehicle is configured for data transmission to a control center, such as a control center located remotely from the vehicle, particularly wherein the system is according to S64 and wherein particularly.V5. The vehicle according to the preceding embodiment, wherein the vehicle comprises a data-transmission component, and wherein the on-board data-processing system is configured for data transmission to the remote data-processing system by means of the data-transmission component of the vehicle.V6. The vehicle according to any of the two preceding embodiments, wherein the vehicle is configured for being controlled from the control center.V7. The vehicle according to any of the preceding vehicle embodiments, wherein the vehicle is retrofitted with the system.V8. The vehicle according to any of the preceding vehicle embodiments, wherein the vehicle is configured for night missions.

[0155] Exemplary features of the invention are further detailed in the figures and the below description of the figures.Brief description of the figuresFig. 1 exemplifies a high-resolution sensor moduleFig. 2 exemplifies a first-type image sensing moduleFig. 3 exemplifies a second-type image sensing moduleFig. 4 exemplifies a head-mounted display (HMD)Fig. 5 exemplifies different views of a vehicle comprising a systemFig. 6A exemplifies an output of an HMDFig. 6B exemplifies a floating window and / or user's viewpoint and the backgroundFig. 6C exemplifies elements in the virtual surface layerFig. 6D exemplifies a movement of a centre point of a floating window to an interactive zoneFig. 6E exemplifies a movement of a centre point of the floating window into an activation zoneFig. 6F exemplifies details of the floating windowFig. 6G exemplifies a viewpoint for a multiple virtual screen solutionFig. 7 exemplifies another output of an HMDFig. 8 exemplifies another view of the systemDetailed figure description

[0156] For the sake of clarity, some features may only be shown in some figures, and others may be omitted. However, also the omitted features may be present, and the shown and discussed features do not need to be present in all embodiments.

[0157] Fig. 5 shows different views of a vehicle 10 comprising a system. The system is a system for improved situational awareness. In the non-limiting example of the discussed figure, the vehicle is an infantry fighting vehicle. However, as discussed above, the vehicle may also be an offroad vehicle, a heavy truck or another military vehicle, e.g., an APC, an engineering platform or a main battle tank. Such vehicles often suffer from poor visibility of the surroundings for the crew and passengers.

[0158] The system comprises a data-processing system, in the example of Fig. 5, an onboard data-processing system 14. The on-board data-processing system 14 is configured to be mounted to a vehicle. In particular, it is configured to be powered by an on-board power network of the vehicle 10.

[0159] The exemplary system of Fig. 5 comprises four image sensing modules. A high- resolution sensor module 20 is oriented towards a front direction of the vehicle 10. The high-resolution sensor module 20 in Fig. 5 is suitable to generate high-resolution image data, which allow for good view for a driver of the vehicle in a direction of drive. The high- resolution sensor module 20 comprises at least one long-wavelength infrared sensor configured for generating thermal image data. Further, the high-resolution sensor module 20 comprises a plurality of image sensors, such as electro-optical sensors, which are configured to capture visible light and / or near-infrared radiation configured for generating image data. Thus, the vehicle can also be navigated under poor light conditions, such as fog, in smoky environments or at darkness. However, the person skilled in the art will easily understand that certain fogs are opaque to infrared radiation, too, and may thus also obstruct the view of the long-wave infrared sensor and / or the image sensors.

[0160] The exemplary system of Fig. 5 further comprises two first-type image sensing modules 30a, 30b which are oriented towards a left and a right side of the vehicle 10, providing for lateral vision. The first-type image sensing modules 30a, 30b are further discussed with respect to Fig. 2.

[0161] The system also comprises a second-type image sensing module 40 oriented in a rear-direction of the vehicle. The second-type image sensing module 40 is further discussed with respect to Fig. 3. In the example of Fig. 5, the second-type image sensing module 40 provides vision in a rear direction and comprises a long-wave-length infrared sensor as well as an image sensor. Thus, optionally advantageously, in all directions of drive, thermal vision is additionally provided, resulting in improved vehicle navigation under poor light conditions.

[0162] Further, the system comprises a plurality of head-mounted displays (HMDs) 50a, 50b. The head-mounted displays may also be helmet-mounted displays. The data- processing system, in the example of Fig. 5 the on-board data-processing system 14, is configured for generating processed image data for each of the HMD(s) based on the image data and the thermal image data, which form the outside image data set. The processing may for example comprise common storing, compressing, buffering or merging.

[0163] Additionally, the data-processing system is configured for deriving, for the HMDs 50a, 50d in Fig. 5, image data from the outside image data set corresponding to a substantially unobstructed view of an environment of the vehicle corresponding to the relative orientation, particularly relative pose, of the HMD.

[0164] In other words, a user of the HMD 50a inside the vehicle 10 gets an output similar to what he or she would see if the vehicle was transparent. The person skilled in the art will easily understand that this output can be enhanced by the thermal image data. Also, the person skilled in the art will easily understand that the output may still be different from the exact view that the user would have if the vehicle was transparent due to blind spots of the image sensing modules or optical distortions. Notwithstanding that, deriving said image data may optionally advantageously improve a situational awareness of the user inside the vehicle 10, as they can easily gain an overview of the surroundings of the vehicle 10.

[0165] In the example of Fig. 5, the data-processing system, particularly the on-board data-processing system 14, is configured for performing a corresponding method of receiving the image data and the thermal image data, deriving image data corresponding to a substantially unobstructed view of an environment of the vehicle corresponding to the relative orientation, particularly relative pose, of the HMD 50a, 50b, and sending the derived image data to the respective HMD.

[0166] Fig. 1 shows an example of the high-resolution sensor module 20. The high- resolution sensor module 20 of Fig. 1 comprises three infrared sensors configured for sensing long-wavelength infrared radiation. Further, the high-resolution sensor module 20 of Fig. 1 comprises three image sensors, such as electro-optical image sensors configured for sensing at least one of visible light and near-infrared radiation, that is, electro-magnetic radiation comprising a wavelength of about 0.75-1.4 pm.

[0167] Further, as can be seen in Fig. 1, to improve visibility, the high-resolution sensor module of Fig. 1 comprises a wiper. Additionally, it is equipped with a cleaning component that can apply a water jet and compressed air for cleaning of the outer covers of the image sensors and the infrared sensors.

[0168] Fig. 2 shows an example of the first-type image sensing module 30. The first-type image sensing module 30 comprises an image sensor. In the non-limiting example of Fig. 2, the first-type image sensing module 30 does not comprise an infrared sensor, thus reducing an amount of generated data.

[0169] Fig. 3 shows an example of the second-type image sensing module 40. In the example of Fig. 3, the second-type image sensing module 40 comprises an infrared sensor and an image sensor. Similar to the high-resolution sensor module 20, the second-type image sensing module 40 in Fig. 3 comprises a wiper as well as a cleaning component configured for using compressed air and / or a water jet for cleaning the outer covers of the image sensor and the infrared sensor. The second-type image sensing module 40 may be suitable for generating image data and thermal image data relating to a direction of drive, for example a rear direction of drive, without generating too much data to be processed. In some embodiments, the second-type image sensing module 40 may also be used for generating image data and thermal image data relating to a forward direction of the vehicle 10, e.g., in case of remote operation, optionally advantageously providing for a good tradeoff of image resolution and image details, still allowing for navigating the vehicle based on the imagery even despite limited bandwidth.

[0170] Fig. 4 shows a head-mounted display (HMD) 50. The HMD may be configured to be connected to the data-processing system by means of a wired connection, e.g., to the onboard data-processing system 14. HMDs for use inside or in close proximity of the vehicle 10 may be referred to as vehicle HMDs.

[0171] The HMD 50 may however also be connected to a remote data-processing system, i.e., a part of the data-processing system remote from the vehicle 10, e.g., in a control center. Such an HMD may be referred to as remote HMD.

[0172] The HMDs 50 may be virtual reality (VR) headsets. The HMDs 50 may comprise a headset that and one screen for each eye. Further, the HMDs 50 comprise at least onesensor for determining an orientation, particularly a pose of a head of the user. For this purpose, the HMDs 50 may comprise a 6-degree-of-freedom inertial measurement unit (6DOF IMU) and / or a gyro sensor. Alternatively or additionally, the HMDs 50 may comprise optical sensors to detect an outside of the HMD.

[0173] In the example of Fig. 5, each HMD 50a, 50b comprises a 6 DOF IMU. Further, the vehicle 10 comprises a 6 DOF IMU. Based on the data of these IMUs, the relative orientation, particularly the relative pose, of the HMDs 50a, 50b with respect to the vehicle 10 is determined.

[0174] Fig. 6A shows a schematic output of the HMDs 50a, 50b of Fig. 5. Fig. 7 shows a concrete example thereof.

[0175] The area extending from a left end to a right end of a screen 61 shows the output that corresponds to a chosen perspective. In the example of Fig. 7, this is a main perspective in a direction of view of the user. The upper area above the chosen perspective shows the main perspective, regardless of the user's selection. The areas left and right of the main area above the chosen perspective show perspectives opposite to a direction of move of the vehicle, similar to virtual back-mirrors of the vehicle 10. In a preferred embodiment, these perspectives are arranged in three respective windows 63.

[0176] Image data 60 can be represented by the screen 61 and the windows 63.

[0177] Further indicators may be displayed, e.g., a danger indicator as rectangular shape around the chosen perspective, or indicators relating to other units or positions.

[0178] An area between the chosen perspective and the upper left, upper middle and upper right area may be a trigger area. Upon activation thereof, the respective perspective may be chosen.

[0179] In the embodiments shown, a floating window 70 can be provided that can represent in this embodiments the danger indicator.

[0180] The HMDs may comprise an eye-tracking mechanism, which may be configured for detecting activation of the trigger areas.

[0181] Fig. 6B exemplifies again the multi-layer information concept of the present invention. Two or more layers of data could represent: a floating window 70 depending on a user's head motion with a view point and a virtual surface that represents a virtual background. The virtual background can comprise an abstract space that mimics some geometrical shape e.g. sphere, where different information is displayed or projected. This enables the user to look, e.g., at the dome surface information through viewpoint and add different information to the viewpoint layer that could compliment the data. In the exampleshown, the virtual background comprises the screen 61 and the windows 63. Further, preferably, an interactive zone 61A is located at an end, preferably at each opposite side ends, of the screen.

[0182] A floating window is represented by the box A. All the grey area constitutes the background area B. As can be seen, the floating window A covers in the example shown the central part of the background B and even some other areas (not being grey in the example shown).

[0183] A user interface with its all features enables a very effective use of the system in following areas, but not limited to - crewed naval, air and land platforms, Uncrewed or optionally crewed naval, air and land platforms and virtual command stations used for mixed domain or multiple platform or other origin data for human representation.

[0184] Fig. 6C shows the elements in the virtual surface layer. Area A represents the largest information that can contain video feed or multiple virtual screens that are most important for the user. The area or field B indicates an interaction element or zone B where this zone enables a user to interact or give commands inside the system. The area of field C is marking one of, e.g., 3 top row windows. This row enables to present more different data to the user. The top row windows have an activations zone D and then they expand to the size presented by window E. Any of the top row 3 screens can be expanded like that. Area F presents an information bar or multipole bars that are in fixed position in relation to the screen A. Letter G indicates a sub-element of the information bar.

[0185] According to the example shown in Fig. 6D, the system is configured to allow the user moving his floating window centre point to the interaction element or zone B, This for example commands the background surface (Fig. 6C) to rotate in the intended direction. This enables buttonless operation of potentially turning the virtual surface 360 degrees around the user.

[0186] Fig. 6E illustrates that the system can allow the user to move the floating window (see Fig. 6B) into the activation zone in the background (see also Fig. 6C). Once a user reaches the activation zone then the window expands for more and clearer information display.

[0187] Fig. 6F exemplifies some sub-elements in the floating window. The floating window comprises information zones A and D, that can contain graphical or text or other type of data to present relevant information to the user.

[0188] Area B indicates the main viewpoint through what the user can see of the background layer. The user can change the field of view value for seeing the background layer. Also, area B shows the tracking for other users. In case other users are in the samefield of view then their current position is visible. Once users are outside of the field of view then indicators are displayed that give information about the users viewing direction relative to the user. The object C indicates the viewpoint centre point and is visible to the user. Box E an be an information bar that can show objects, texts or other data. Zone H can be used for indicating presence of relevant data outside of user current viewpoint e.g. friendly forces tracking information. G indicates to the data that can be displayed and also tracked on this object E. Area F is meant for but not limited to addition tracking field that might indicate to the position of some mission critical system (e.g. a weapon).

[0189] Fig. 6G exemplifies a conceptional layout of a command station. This example is illustrating the viewpoint for the multiple virtual screen solution. A is the viewpoint window that the user controls to observe background virtual space B, where the different combination layout of multiple screens or other virtual objects are displayed. The rest of the interaction and information field concept remains similar to the main product.Improvement of the headtracking

[0190] In addition to using IMU based headtracking and different algorithmic corrections the system also reduce IMU drifting by stabilizing the yaw and pitch angles tracked by the IMUs in the goggles. By clamping extreme rotations and adjusting for drift, the code ensures smoother and more accurate head movements. This keeps virtual objects and the live video feed properly aligned with the user's real-world movements, enhancing the overall experience. In addition, one or more of the 3 axes could be selectively turned off from tracking sequence.

[0191] The output may comprise further information, e.g., controls relating to the vehicle, as can be seen in Fig. 7 below the chosen perspective. Thus, optionally advantageously, a more cumbersome augmented reality solution allowing to the user to additionally see actual, physical controls of the vehicle 10 despite wearing the HMD may not be necessary.

[0192] Fig. 8 shows another example of the system. In the example of Fig. 8, the system further comprises the remote data-processing system 16. The on-board data-processing system 14 may be configured for transmitting image data to the remote data-processing system 16. The remote data-processing system is connected, in the example of Fig. 8, to another two HMDs 50c, 50d. These two HMDs may be referred to as remote HMDs, as discussed above. Optionally advantageously, thus, remote control and / or remote supervision of the vehicle may be enabled.

[0193] Similar to the example of Fig. 8, the data-processing system, particularly the onboard data-processing system 14, is configured for performing a corresponding method ofreceiving the image data and the thermal image data. Further, the data-processing system derives the image data corresponding to the orientation and / or pose of the remote HMDs and outputs these image data via the remote HMDs. Additionally, the method comprises the on-board data-processing system 14 transmitting image data, the image data set or a part of the image data set to the remote data-processing system 16. In the example of Fig. 8, the connection between the on-board data-processing system 14 and the remote data-processing system 16 is facilitated by a transmission network component 18, such as a radio station connected to the remote data-processing system, a satellite communication network or a mobile phone network.

[0194] The data processing system, particularly the on-board data-processing system 14 and / or, where present, the remote data-processing system 16, may comprise one or more processing units configured to carry out computer instructions of a program (i.e. machine readable and executable instructions). The processing unit(s) may be singular or plural. For example, the data processing system may comprise at least one of CPU, GPU, DSP, APU, ASIC, ASIP or FPGA. The data processing system may comprise memory components, such as, main memory (e.g. RAM), cache memory (e.g. SRAM) and / or secondary memory (e.g. HDD, SDD). The data processing system may comprise volatile and / or non-volatile memory such an SDRAM, DRAM, SRAM, Flash Memory, MRAM, F-RAM, or P-RAM. The data processing system may comprise internal communication interfaces (e.g. busses) configured to facilitate electronic data exchange between components of the data processing system, such as, the communication between the memory components and the processing components. The data processing system may comprise external communication interfaces configured to facilitate electronic data exchange between the data processing system and devices or networks external to the data processing system, such as a bus network of the vehicle 10. For example, the data processing system may comprise network interface card(s) that may be configured to connect the data processing system to a network, such as, to the Internet or the network of the vehicle 10. The data processing system may be configured to transfer electronic data using a standardized communication protocol. The data processing system may be a centralized or distributed computing system. In particular, the remote data-processing system 16 may be a distributed computing system.

[0195] The data-processing system may be configured for data exchange between the onboard data-processing system 14 and the remote data-processing system 16, for example by use of a data-transmission system of the vehicle 10 and further data-transmission systems.

[0196] The data processing system may comprise user interfaces, such as: output user interface, such as:o screens or monitors configured to display visual data (e.g. displaying graphical user interfaces of the questionnaire to the user), o speakers configured to communicate audio data (e.g. playing audio data to the user), input user interface, such as: o camera configured to capture visual data (e.g. capturing images and / or videos of the user), o microphone configured to capture audio data (e.g. recording audio from the user), o keyboard configured to allow the insertion of text and / or other keyboard commands (e.g. allowing the user to enter text data and / or other keyboard commands by having the user type on the keyboard) and / or o trackpad, mouse, touchscreen, joystick - configured to facilitate the navigation through different graphical user interfaces of the questionnaire.

[0197] To put it simply, the data processing system may be a processing unit configured to carry out instructions of a program. The data processing system may comprise a system- on-chip comprising processing units, memory components and busses. The data processing system may be a personal computer, a laptop, a pocket computer, a smartphone, a tablet computer. The data processing system may comprise a server, a server system, a portion of a cloud computing system or a system emulating a server, such as a server system with an appropriate software for running a virtual machine. The data processing system may be a processing unit or a system-on-chip that may be interfaced with a personal computer, a laptop, a pocket computer, a smartphone, a tablet computer and / or user interfaces (such as the upper-mentioned user interfaces).

[0198] While in the above, a preferred embodiment has been described with reference to the accompanying drawings, the skilled person will understand that this embodiment was provided for illustrative purpose only and should by no means be construed to limit the scope of the present invention, which is defined by the claims.

[0199] Whenever a relative term, such as "about", "substantially" or "approximately" is used in this specification, such a term should also be construed to also include the exact term. That is, e.g., "substantially straight" should be construed to also include "(exactly) straight".

[0200] Whenever steps were recited in the above or also in the appended claims, it should be noted that the order in which the steps are recited in this text may be accidental. Thatis, unless otherwise specified or unless clear to the skilled person, the order in which steps are recited may be accidental. That is, when the present document states, e.g., that a method comprises steps (A) and (B), this does not necessarily mean that step (A) precedes step (B), but it is also possible that step (A) is performed (at least partly) simultaneously with step (B) or that step (B) precedes step (A). Furthermore, when a step (X) is said to precede another step (Z), this does not imply that there is no step between steps (X) and (Z). That is, step (X) preceding step (Z) encompasses the situation that step (X) is performed directly before step (Z), but also the situation that (X) is performed before one or more steps (Yl), ..., followed by step (Z). Corresponding considerations apply when terms like "after" or "before" are used.Numbered reference signs10 Vehicle14 On-board data-processing system16 Remote data-processing system 18 Transmission network component20 High-resolution sensor module30, 30a, 30b First-type image sensing module(s)40 Second-type image sensing module(s)50, 50a, 50b, 50c, 50d HMD(s) 60 Image data61 Screen62 Thermal image data63 Windows70 Floating window

Claims

Claims1. A system for processing image information configured to be connected to a vehicle, particularly a combat vehicle, the system comprising a plurality of image sensing modules configured for sensing an outside image data set, the plurality of image sensing modules comprising o a high-resolution sensor module, o at least one or a plurality of first-type image sensing module(s), o at least one second-type image sensing module,- a data-processing system configured for o receiving the outside image data set from the image sensing modules, and o processing the outside image data set.

2. The system according to the preceding claim, wherein the high-resolution sensor module comprises at least two infrared sensors, particularly at least three infrared sensors, which infrared sensors are configured for generating thermal image data, wherein the high- resolution sensor module comprises at least two image sensors, particularly at least three image sensors, which image sensors are configured for generating image data based on at least one of visible light and near-infrared radiation, wherein each first-type image sensing module comprises an image sensor configured for generating image data based on visible light, wherein the second-type image sensing module comprises an infrared sensor, which infrared sensor is configured for generating thermal image data, and wherein the second- type image sensing module comprises an image sensor, which image sensor is configured for generating image data based on at least one of visible light and near-infrared radiation.

3. The system according to any of the preceding claims, wherein the system comprises at least three HMDs, particularly four HMDs, and wherein the HMD(s) are configured for receiving image data from the data- processing system.

4. The system according to the preceding claim, wherein the system is configured for determining a relative orientation, particularly a relative pose, of the HMD(s) with respect to the vehicle.

5. The system according to any of the two preceding claims, wherein the system is configured to be integrated into and / or mounted to the vehicle, particularly the combat vehicle, wherein the data-processing system is configured for generating processed image data for each of the HMD(s) based on the processed outside image data set, wherein the system is further configured for determining a relative orientation ofeach HMD with respect to the vehicle, particularly a relative pose of each HMD with respect to the vehicle, and wherein the data-processing system is configured for generating the processed image data for each HMD by deriving image data from the outside image data set corresponding to a substantially unobstructed view of an environment of the vehicle corresponding to the relative orientation, particularly relative pose, of the HMD.

6. The system according to the preceding claim, wherein the data-processing system is configured for multi-view rendering.

7. The system according to any of the two preceding claims, wherein generating the processed image data for each of the HMD(s) based on the outside image data set comprises providing at least one image section relating to a rear perspective and / or a side perspective, wherein particularly, the system is configured for enlarging the rear perspective and / or the side perspective shown by an HMD upon receiving an instruction of a user wearing the HMD.

8. The system according to any of the preceding claims but claims 5-7, wherein the data- processing system comprises the on-board data-processing system and a remote data- processing system, wherein the on-board data-processing system is configured to be mounted to or integrated into the vehicle, wherein the remote data-processing system is located remote from the vehicle, and wherein the on-board data-processing is configured for transmission of at least a part of the image data set to the remote data-processing system, wherein at least one or a plurality of the HMD(s) are remote HMD(s), wherein the remote HMD(s) are configured for receiving data from the remote data-processing system, wherein the system is further configured for determining an absolute orientation of the remote HMD(s), particularly an absolute pose of the remote HMD(s).

9. The system according to the preceding claim, wherein the data-processing system, particularly the remote data-processing system, is configured for generating the processed image data for the remote HMD(s) by deriving image data from at least the part of the outside image data corresponding to the substantially unobstructed view of an environment of the vehicle based on the absolute orientation, particularly based on the absolute pose, of the remote HMD(s).

10. The system according to any of the preceding claims with the features of claim 2, wherein at least one of processing the outside image data set and generating processed the image data for each of the HMD(s) comprises fusing the thermal image data and the image data.

11. The system according to any of the preceding claims, wherein the image sensing modules are configured to be placed on the vehicle to cover a horizontal field of view of at least 270°, preferably 315°, still more preferably 360° around the vehicle.

12. The system according to any of the preceding claims, wherein each of the image sensing modules comprises a cleaning component, wherein each image sensing module comprises a connection to a water provision system and wherein each cleaning component of the image sensing modules comprises a water jet cleaning component for cleaning the module's sensor(s).

13. The system according to any of the preceding claims, wherein the high-resolution sensor module comprises at least one wiper, and wherein the second-type image sensing module comprises at least one wiper.

14. A vehicle, comprising the system according to any of the preceding claims.

15. The vehicle according to the preceding claim, wherein the vehicle is at least one of an offroad vehicle, a heavy vehicle and an armored vehicle.

16. The vehicle according to any of the two preceding claims, wherein the vehicle is configured for night missions.

Images