Detecting the surroundings of and tele-operated guidance of an ego vehicle

Abstract

According to a method for detecting the surroundings, a first image data stream (15) is generated by means of a first camera (2) of an ego vehicle (4) and a second image data stream is received which is generated by means of a second camera of a further vehicle (5), wherein a visual field (8) represented by the first image data stream (15) overlaps with a second visual field (9) represented by the second image data stream. A region (14) which is obscured for the first camera (2) is identified in the first visual field (8), which visual field is not obscured for the second camera. On the basis of the second image data stream, substitute image data (17) are generated which correspond to the region (14) which is obscured for the first camera (2). On the basis of the first image data stream (15), a combined image data stream is generated which represents the first visual field, wherein in a region (14) of the combined image data stream (16), which corresponds to the region (14) obscured for the first camera (2), the substitute image data (17) are displayed.

Description

[0001] Environment detection and teleoperated control of an ego vehicle

[0002] The present invention relates to a method for environmental detection, wherein a first image data stream is generated by means of a first camera of an ego vehicle, as well as to a method for teleoperated control of an ego vehicle. The invention further relates to a corresponding environmental detection system, a vehicle guidance system for teleoperated control of an ego vehicle, and a computer program product.

[0003] If a highly automated vehicle, particularly a fully autonomous vehicle, encounters a situation that the vehicle can no longer handle independently or can no longer handle with sufficient safety, so-called teleoperated guidance can be used. This involves a human teleoperator accessing the data from the vehicle's surroundings, collected by various sensor systems, particularly cameras, via an external computer system. They can then remotely control the vehicle to resolve the situation.

[0004] In some situations, however, the field of view of a vehicle's camera is partially obscured, for example, by another vehicle ahead. The teleoperator may then fail to take relevant but obscured information into account during remote control, which can pose a safety risk.

[0005] It is an object of the present invention to increase safety during teleoperated driving of a vehicle.

[0006] This object is achieved by the respective subject matter of the independent claims. Advantageous further developments and preferred embodiments are the subject matter of the dependent claims.

[0007] The invention is based on the idea of ​​overcoming limitations in the field of view of the camera of the leading vehicle caused by obscuring objects, such as other vehicles driving ahead, by combining an image data stream from a camera of another vehicle with the image data stream of the leading vehicle. In particular, substitute image data from the image data stream of the other vehicle is displayed in a concealed area of ​​the image data stream of the leading vehicle.

[0008] According to one aspect of the invention, a method for environmental detection is specified, wherein a first image data stream is generated by means of a first camera of an ego vehicle. A second image data stream is obtained, generated by means of a second camera of a further vehicle, in particular by means of at least one computing unit. A first field of view represented by the first image data stream overlaps with a second field of view represented by the second image data stream. Based on the first image data stream and the second image data stream, an area in the first field of view that is hidden from the first camera and not hidden from the second camera is identified, in particular by means of the at least one computing unit. Based on the second image data stream, substitute image data is generated that corresponds to the area hidden from the first camera, in particular by means of the at least one computing unit.Based on the first image data stream, and in particular the second image data stream, a combined image data stream is generated that represents the first field of view. The substitute image data is displayed in a region of the combined image data stream that corresponds to the area occluded for the first camera.

[0009] The second image data stream is obtained in particular by means of the at least one computing unit, for example by means of at least one vehicle computing unit of the ego vehicle, in particular via a first communication interface for wireless communication, for example V2V or V2X communication. V2V stands for vehicle-to-vehicle, and V2X stands for vehicle-to-anything.

[0010] The at least one computing unit contains, for example, the at least one vehicle computing unit and can, for example, include a vehicle-external computer system, i.e., a computer system arranged externally to the ego vehicle and the other vehicle. The vehicle-external computer system can, for example, be referred to as a server computer system or backend computer system. In particular, it can be a computer system for the remote control of vehicles, in particular the ego vehicle.The second image data stream is generated in particular by means of the second camera and transmitted from the further vehicle, for example a further vehicle computing unit of the further vehicle, in particular via a further communication interface for wireless communication, for example for V2V or V2X communication, to the ego vehicle, in particular the at least one computing unit, for example the at least one vehicle computing unit, in particular via the first communication interface.

[0011] An image data stream can be understood, for example, as a sequence of two or more consecutive images. The first image data stream, in particular corresponding first images of the first image data stream, depicts, in particular, an external environment of the ego vehicle. The first field of view, which can also be understood as the field of view of the first camera, thus lies, in particular, at least partially, in the external environment of the ego vehicle. The second image data stream, in particular corresponding second images of the second image data stream, depicts, in particular, an external environment of the further vehicle. The second field of view, which can also be understood as the field of view of the second camera, thus lies, in particular, at least partially, in the external environment of the further vehicle.

[0012] Since the first field of view and the second field of view overlap, the first and second image streams generally represent partially identical content. However, since the area occluded for the first camera is not occluded for the second camera, the second image stream represents content that the first image stream does not represent for the first camera due to the occlusion. For example, the occlusion can be caused by an occluding object in the first field of view. However, the occluding object is not necessarily the other vehicle itself.

[0013] The substitute image data can therefore be generated, for example, based on those image data of the second image data stream that represent a region of the second field of view that also lies in the first field of view and that would be represented by the first image data stream if the obscuring object did not exist. The corresponding image data of the second image data stream can, in particular, be processed or modified to generate the substitute image data, in particular to adapt the representation to the first image data stream. The combined image data stream corresponds, in particular, to a combination of the first image data stream or a processed variant of the first image data stream with the substitute image data.

[0014] Outside the area corresponding to the area hidden from the first camera, the combined image data stream can, for example, be the same as the first image data stream. In the area corresponding to the area hidden from the first camera, the combined image data stream can, for example, correspond to the substitute image data or an overlay of the substitute image data with the first image data stream. A transition area can also be provided between the aforementioned areas.

[0015] The combined image data stream therefore contains information through the substitute image data that would otherwise be unavailable to the ego vehicle due to the occlusion. A viewer of the combined image data stream, such as a teleoperator who remotely controls the ego vehicle via teleoperated guidance, can therefore be more comprehensively informed by the ego vehicle about the current conditions in the ego vehicle's surroundings, which ultimately increases safety during teleoperated guidance of the ego vehicle.

[0016] According to at least one embodiment of the method for environmental detection according to the invention, a deviation of one or more extrinsic camera parameters and / or one or more intrinsic camera parameters between the first camera and the second camera is at least partially compensated by transforming the second image data stream according to a, for example, predetermined, transformation parameter set, in particular by means of the at least one computing unit, for example by means of the at least one vehicle computing unit. The area hidden from the first camera is identified based on the first image data stream and the transformed second image data stream, in particular by means of the at least one computing unit, for example by means of the at least one vehicle computing unit.

[0017] Extrinsic camera parameters include, for example, a pose—that is, a position and orientation—of a camera coordinate system of the corresponding camera relative to a reference coordinate system. The camera coordinate system is rigidly connected to the respective camera. The reference coordinate system can, for example, be rigidly connected to the respective vehicle or to the surroundings of the respective vehicle. A common reference coordinate system can also be selected for cameras of different vehicles. The pose can, for example, be defined by six parameters, such as three position parameters and three orientation parameters. The three position parameters can, for example, be three-dimensional coordinates of a coordinate origin of the camera coordinate system in the reference coordinate system.The three orientation parameters can be, for example, three orientation angles, such as Euler angles, which can also be called pitch, yaw and roll angles.

[0018] The intrinsic camera parameters can include parameters that define how a point in the camera's surroundings is imaged by the camera, in particular on an image sensor of the camera. The intrinsic camera parameters can therefore, for example, describe or define an imaging function of a camera lens, a focal length, and so on. The intrinsic camera parameters can also include an aspect ratio, a resolution, and / or a pixel density of the image sensor. The intrinsic camera parameters can also relate to camera-internal preprocessing of the captured image data, for example, parameters relating to white balance, image sharpening, color correction, or the like.

[0019] According to at least one embodiment, the combined image data stream is generated based on the first image data stream and the transformed second image data stream, wherein the replacement image data is generated based on the transformed second image data stream, in particular by means of the at least one computing unit, for example by means of the at least one vehicle computing unit.

[0020] By at least partially compensating for the deviation of the extrinsic and / or intrinsic camera parameters, deviations between the first image data stream and the second image data stream in a scale and / or scaling of imaged objects are at least partially compensated, an offset and / or a shift of the first field of view with respect to the second field of view is at least partially compensated, and / or a perspective deviation between the first image data stream and the second image data stream is at least partially compensated. This ensures that the substitute image data, in combination with the image data from the first image data stream, allows a consistent representation of the surroundings, which ultimately further increases safety in the application of teleoperated driving of the ego vehicle.According to at least one embodiment, at least one feature is identified, in particular based on the first image data stream and the second image data stream, which is represented by both the first image data stream and the second image data stream, in particular by means of the at least one computing unit, for example by means of the at least one vehicle computing unit. The transformation parameter set is determined as a function of a comparison of a representation of the at least one feature in the first image data stream with a representation of the at least one feature in the second image data stream, in particular by means of the at least one computing unit, for example by means of the at least one vehicle computing unit.

[0021] By identifying and comparing the aforementioned feature, the effects of the deviation of one or more extrinsic camera parameters and / or one or more intrinsic camera parameters between the first camera and the second camera can be extracted directly from the image data streams. Deviations in the camera parameters that may not have a relevant impact are thus advantageously not explicitly considered and therefore do not need to be explicitly determined. This makes the compensation of the deviation in the camera parameters more efficient.

[0022] According to at least one embodiment, a tonal value distribution and / or a color value distribution of the second image data stream is compared with a tonal value distribution and / or a color value distribution of the first image data stream, in particular by means of the at least one computing unit, for example by means of the at least one vehicle computing unit. Based on a result of the comparison, the transformed second image data stream is corrected, or the second image data stream is corrected before the transformation. The substitute image data is generated based on the transformed and corrected second image data stream.

[0023] This ensures, in particular, that the brightness and white balance in the combined image data stream are consistent, which ultimately further increases safety in the application of teleoperated driving of the ego vehicle. The tonal value distribution and / or a color value distribution can be provided, for example, by one or more corresponding histograms.

[0024] According to at least one embodiment, the substitute image data are superimposed on the first image data stream, in particular on the region of the first image data stream corresponding to the region hidden for the first camera, in order to generate the combined image data stream.

[0025] For example, the replacement image data are partially transparently overlaid on the original image data of the first image data stream, which corresponds to the area in the first image data stream that is hidden from the first camera.

[0026] The combined image data stream thus displays both the substitute image data and the original image data. The combined image data stream thus represents both the occluding object and content located behind the occluding object as seen from the first camera. The observer, for example, the teleoperator, can therefore consider more information when making decisions, which ultimately further increases safety in the application of teleoperated driving of the ego vehicle.

[0027] According to at least one embodiment, the additional vehicle is located in the first field of view. In particular, the additional vehicle is represented by the first image data stream, and the area obscured from the first camera is obscured from the first camera by the additional vehicle. In other words, the additional vehicle is the obscuring object.

[0028] For example, the additional vehicle may be driving ahead of the ego vehicle on a roadway, in particular in the same lane as the ego vehicle or in another, for example, adjacent, lane. The first camera may, for example, be a camera oriented in the forward direction of travel of the ego vehicle, also referred to as a front camera. The second camera may, for example, be a front camera of the additional vehicle. According to at least one embodiment, the area obscured from the first camera is identified by means of the at least one vehicle processing unit of the ego vehicle.

[0029] Alternatively, the at least one vehicle computing unit can transmit the first image data stream and the second image data stream to the external computer system, in particular via the first communication interface or a second communication interface, and the external computer system can identify the area hidden from the first camera.

[0030] According to at least one embodiment, the substitute image data are determined by means of the at least one vehicle computing unit of the ego vehicle.

[0031] Alternatively, the at least one vehicle computing unit can transmit the second image data stream and optionally the first image data stream to the external computer system, in particular via the first communication interface or the second communication interface, and the external computer system can determine the replacement image data.

[0032] According to at least one embodiment, the combined image data stream is generated by means of the at least one vehicle computing unit of the ego vehicle.

[0033] Alternatively, the at least one vehicle computing unit can transmit the first image data stream to the external computer system, in particular via the first communication interface or a second communication interface. The at least one vehicle computing unit can also transmit the second image data stream or the substitute image data to the external computer system. The external computer system can then generate the combined image data stream.

[0034] In a preferred embodiment, the steps of identifying the obscured area, generating the substitute image data, and generating the combined image data stream are performed by the at least one vehicle computing unit. For example, the combined image data stream is transmitted to the external computer system by the at least one vehicle computing unit, in particular via the first communication interface or the second communication interface. According to at least one embodiment, the combined image data stream is displayed by the vehicle-external computer system on a vehicle-external display device, such as one or more screens or monitors or a head-mounted visual output device, also referred to as an HMD (head-mounted display).Previously, the combined image data stream is generated, for example, by means of the at least one vehicle computing unit and transmitted, in particular via the first communication interface or the second communication interface, to the external computer system, or the combined image data stream is generated by means of the external computer system.

[0035] According to a further aspect of the invention, a method for teleoperated driving of an ego vehicle is specified. For this purpose, a method according to the invention for environmental detection is carried out, wherein in particular the combined image data stream is displayed on the vehicle-external display device by means of the vehicle-external computer system. In response to the display of the combined image data stream on the vehicle-external display device, i.e. in particular after the display or simultaneously with the display, a user input, in particular from the teleoperator, is detected by means of the vehicle-external computer system. The ego vehicle is driven at least partially automatically, in particular remotely, depending on the user input.

[0036] According to at least one embodiment of the method for teleoperated control of the ego vehicle, a control command is generated by the vehicle-external computer system depending on the user input and transmitted to the ego vehicle. The ego vehicle is controlled at least partially automatically depending on the control command, in particular by means of an ego vehicle control system of the ego vehicle.

[0037] The ego-vehicle guidance system can be understood, in particular, as an electronic system designed to guide the ego-vehicle fully automatically or fully autonomously, in particular without requiring driver intervention. The ego-vehicle automatically performs all required functions, such as steering, braking, and / or acceleration maneuvers, monitoring and detecting road traffic, and corresponding reactions. In particular, the ego-vehicle guidance system can implement a fully automatic or fully autonomous driving mode of the motor vehicle according to Level 5 of the classification according to SAE J3016. Alternatively, the ego-vehicle guidance system can implement a highly automated driving mode of the motor vehicle according to Level 4 of the classification according to SAE J3016. Here and below, "SAE J3016" refers to the corresponding standard in the June 2018 version.

[0038] The ego vehicle guidance system can, for example, generate one or more control signals depending on the control command and transmit them to one or more actuators of the ego vehicle so that they influence or carry out lateral and / or longitudinal control of the ego vehicle according to the control command.

[0039] According to a further aspect of the invention, an environment detection system is specified which has a first camera for an ego vehicle, which is configured to generate a first image data stream. The environment detection system has at least one communication interface for the ego vehicle for wireless data transmission and at least one computing unit, in particular at least one vehicle computing unit. The at least one computing unit, in particular the at least one vehicle computing unit, is configured to receive a second image data stream generated by a second camera of a further vehicle via the at least one communication interface, for example the first communication interface, wherein a first field of view represented by the first image data stream overlaps with a second field of view represented by the second image data stream.The at least one computing unit, in particular the at least one vehicle computing unit, is configured to identify, based on the first image data stream and the second image data stream, an area in the first field of view that is obscured for the first camera but not obscured for the second camera, and to generate substitute image data based on the second image data stream that corresponds to the area obscured for the first camera. The at least one computing unit, in particular the at least one vehicle computing unit, is configured to generate, based on the first image data stream and in particular the second image data stream, a combined image data stream that represents the first field of view, and to display the substitute image data in an area of ​​the combined image data stream that corresponds to the area obscured for the first camera.According to at least one embodiment of the environment detection system, the environment detection system is designed as an ego vehicle guidance system for the ego vehicle.

[0040] According to at least one embodiment of the environment detection system, it has a vehicle-external display device and the at least one computing unit includes a vehicle-external computer system.

[0041] In such embodiments, the environment detection system may also be referred to as a vehicle guidance system for teleoperated guidance of the ego vehicle.

[0042] According to at least one embodiment of the environment detection system, the vehicle-external computer system is configured to generate the combined image data stream and display it on the vehicle-external display device. The vehicle-external computer system can, for example, receive the first image data stream and the second image data stream from the at least one vehicle computing unit.

[0043] According to at least one embodiment of the environment detection system, the at least one vehicle computing unit is configured to receive the second image data stream via the at least one communication interface, in particular the first communication interface, identify the area obscured for the first camera, generate the substitute image data, generate the combined image data stream, and transmit the combined image data stream to the vehicle-external computer system via the at least one communication interface, in particular the first communication interface or the second communication interface. The vehicle-external computer system is configured to display the combined image data stream on the vehicle-external display device.

[0044] According to at least one embodiment of the vehicle guidance system for teleoperated guidance of the ego vehicle, the vehicle-external computer system is configured to detect a user input in response to the display of the combined image data stream on the vehicle-external display device. The vehicle guidance system for teleoperated guidance comprises an ego vehicle guidance system for the ego vehicle, which is configured to guide the ego vehicle at least partially automatically depending on the user input.

[0045] According to at least one embodiment of the vehicle guidance system for teleoperated guidance of the ego vehicle, the vehicle-external computer system is configured to generate a control command based on the user input and transmit it to the ego vehicle. The ego vehicle guidance system is configured to guide the ego vehicle at least partially automatically based on the control command.

[0046] Further embodiments of the environmental detection system according to the invention and of the vehicle guidance system according to the invention follow directly from the various embodiments of the environmental detection method according to the invention or of the method according to the invention for teleoperated guidance of an ego vehicle, and vice versa. In particular, individual features and corresponding explanations relating to the various embodiments of the methods according to the invention can be transferred analogously to corresponding embodiments of the environmental detection system according to the invention and of the vehicle guidance system according to the invention. In particular, the environmental detection system according to the invention is designed or programmed to carry out an environmental detection method according to the invention. In particular, the environmental detection system according to the invention carries out the environmental detection method according to the invention.In particular, the vehicle guidance system according to the invention is designed or programmed to carry out a method according to the invention for the teleoperated control of an ego vehicle. In particular, the vehicle guidance system according to the invention carries out the method according to the invention for the teleoperated control of an ego vehicle.

[0047] According to a further aspect of the invention, a first computer program with first instructions is specified. When the first instructions are executed by an environment detection system according to the invention, the first instructions cause the environment detection system to carry out an environment detection method according to the invention. According to a further aspect of the invention, a second computer program with second instructions is specified. When the second instructions are executed by a vehicle guidance system according to the invention for the teleoperated guidance of an ego vehicle, the second instructions cause the vehicle guidance system to carry out an inventive method for the teleoperated guidance of an ego vehicle.

[0048] According to a further aspect of the invention, a computer-readable storage medium is provided which stores a first computer program according to the invention and / or a second computer program according to the invention.

[0049] The first computer program, the second computer program and the computer-readable storage medium may be considered as respective computer program products having the first and / or second instructions.

[0050] A computing unit can be understood, in particular, as a data processing device that contains a processing circuit. The computing unit can therefore, in particular, process data to perform computing operations. This may also include operations for performing indexed access to a data structure, for example, a look-up table (LUT).

[0051] The computing unit may, in particular, contain one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more single-chip systems (SoCs). The computing unit may also contain one or more processors, for example one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit may also include a physical or virtual network of computers or other of the aforementioned units.In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0052] A memory unit can be a volatile data memory, for example a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a non-volatile data memory, for example a read-only memory (ROM), a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM), a magnetoresistive random access memory (MRAM), or a phase-change random access memory.PCRAM (phase-change random access memory),

[0053] If, within the scope of the present disclosure, it is stated that a component of the environmental detection system or vehicle guidance system according to the invention, in particular the at least one computing unit, the at least one vehicle computing unit and / or the vehicle-external computer system, is set up, designed, configured, or the like to carry out or implement a specific function, to achieve a specific effect or to serve a specific purpose, this can be understood to mean that the component, beyond the fundamental or theoretical usability or suitability of the component for this function, effect or purpose, is concretely and actually capable of carrying out or implementing the function, achieving the effect or serving the purpose through appropriate adaptation, programming, physical design, and so on.

[0054] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures can be encompassed by the invention not only in the respectively specified combination, but also in other combinations. In particular, the invention can also encompass embodiments and combinations of features that do not have all the features of an originally formulated claim. Furthermore, the invention can encompass embodiments and combinations of features that go beyond the combinations of features set out in the backreferences to the claims or deviate from them.

[0055] The invention is explained in more detail below using specific embodiments and associated schematic drawings. In the figures, identical or functionally equivalent elements may be provided with the same reference numerals. The description of identical or functionally equivalent elements may not necessarily be repeated for different figures.

[0056] Showing:

[0057] Fig. 1 is a schematic representation of an exemplary embodiment of an environment detection system according to the invention; and

[0058] Fig. 2 is a schematic flow diagram of an exemplary embodiment of a method for environmental detection according to the invention.

[0059] Fig. 1 schematically shows an exemplary embodiment of an environment detection system 1 according to the invention, which in particular also represents a vehicle guidance system according to the invention for the teleoperated guidance of an ego vehicle 4.

[0060] The ego vehicle 4 is driving on a roadway behind another vehicle 5. For example, other vehicles 6, 7 can be driving in front of the other vehicle 5.

[0061] The environment detection system 1 has a first camera 2, in particular a front camera, of the ego vehicle 4, which is configured to generate a first image data stream 15 representing a first field of view 8 of the first camera 2. The further vehicle 5 and, for example, the further vehicles 6, 7 are located in the first field of view 8, wherein the further vehicle 5, for example, obscures an area 14 in the first field of view 8 for the first camera 2, so that in particular the further vehicles 6, 7 are partially obscured in the first image data stream 15.

[0062] The further vehicle 5 has a second camera 3, in particular a front camera, which is configured to generate a second image data stream representing a second field of view 9 of the second camera 3. The further vehicles 6, 7 are located, for example, in the second field of view 9 and are, in particular, not obscured or less obscured for the second camera 3 than for the first camera 2.

[0063] The ego vehicle 4 and the additional vehicle 5 each have a communication interface for wireless data transmission, for example, a V2V or V2X interface. Furthermore, the environment detection system 1 has a vehicle computing unit 10 and a vehicle-external computer system 10, which, for example, is part of a backend for remote vehicle control.

[0064] The vehicle computing unit 10 receives the second image data stream from the further vehicle 5, for example from a further vehicle computing unit of the further vehicle 5, via the communication interfaces.

[0065] The vehicle computing unit 10 identifies, for example, based on a comparison of the first image data stream 15 with the second image data stream, the area 14 hidden for the first camera 2 and, based on the second image data stream, generates substitute image data 17 that corresponds to the area 14 hidden for the first camera 2. The vehicle computing unit 10 then generates a combined image data stream 16 that represents the first field of view 8, wherein the substitute image data 17 are displayed in an area of ​​the combined image data stream 16 that corresponds to the area 14 hidden for the first camera 2, for example, they are superimposed semi-transparently on the original image data of the first image data stream 15.

[0066] The vehicle computing unit 10 can wirelessly transmit the combined image data stream 16 to the vehicle-external computer system 13. The environment detection system 1, in particular the backend, has a vehicle-external display device 11. The vehicle-external computer system 13 is configured to generate the combined image data stream 16 and display it on the vehicle-external display device 11. A teleoperator 12 can analyze the displayed combined image data stream 16 and, in response, make a user input on an input device of the vehicle-external computer system 13. Depending on the user input, the vehicle-external computer system 13 can transmit a control command to the vehicle computing unit 10. Based on the control command, the ego vehicle 4 can then be guided at least partially automatically.

[0067] Fig. 2 shows a schematic flow diagram of an exemplary embodiment of a method for environmental detection according to the invention.

[0068] In step SOa, the additional vehicle 5 is identified and selected, for example, based on V2V or V2X capabilities of the additional vehicle 5 and / or the relative position to the ego vehicle 4. Furthermore, the second image data stream is generated and transmitted to the ego vehicle 4. In step SOb, the first image data stream is generated.

[0069] In step S1a, features in the second image data stream are identified, and in step S1b, features in the first image data stream 15 are identified. In step S2, based on the previously identified features, matching features from the first image data stream 15 and the second image data stream are then identified. The features can be, for example, objects or edges in the respective image data stream or the like. In step S3, the matching features are compared, and in step S4, depending on the comparison, a transformation parameter set is determined that approximately converts the matching features of the second image data stream into the matching features of the first image data stream 15. In particular, the transformation parameters of the transformation parameter set are optimized so that an optimal superposition of the matching features is achieved.The transformation parameter set particularly concerns a scale, shifts and / or perspective adjustments.

[0070] Since the first camera 2 and the second camera 3 do not necessarily have the same white balance, brightness, and contrast, the color of the second image data stream can optionally also be adjusted. This can be achieved, for example, by adjusting the histograms of the first image data stream 15 and the second image data stream. In step S6, the obscured area 14 is masked. For this purpose, the relevant areas are identified by comparing both image data streams and significantly different areas. These areas are most likely obscured by the other vehicle 5. Alternatively, the location data of the vehicle ahead can also be used, if available. In step S7, the combined image data stream 16 is generated.

[0071] In known teleoperated vehicles, the teleoperator's view is limited to the camera images of the vehicle being driven. Due to the lack of direct feedback from the vehicle and the surroundings, it is desirable to support the teleoperator as much as possible.

[0072] The invention makes it possible in various embodiments to reduce the restriction of the tele-operator's view caused by obscuring objects by using camera data from other road users.

[0073] In various embodiments of the invention, V2X communication technologies are used, which enable communication between individual vehicles and fixed units for the exchange of sensor data.

Claims

Patent claims 1. A method for environmental sensing, wherein a first image data stream (15) is generated by means of a first camera (2) of an ego-vehicle (4), characterized in that a second image data stream generated by means of a second camera of another vehicle (5) is obtained, wherein a first field of view (8) represented by the first image data stream (15) overlaps with a second field of view (9) represented by the second image data stream; based on the first image data stream (15) and the second image data stream, an area (14) in the first field of view (8) that is obscured for the first camera (2) and is not obscured for the second camera is identified; based on the second image data stream, substitute image data (17) are generated that correspond to the area (14) obscured for the first camera (2);and based on the first image data stream (15) a combined image data stream is generated which represents the first field of view, wherein in an area of ​​the combined image data stream (16) which corresponds to the area (14) obscured for the first camera (2) the substitute image data (17) are displayed.

2. Method according to claim 1, characterized in that a deviation of one or more extrinsic camera parameters and / or one or more intrinsic camera parameters between the first camera (2) and the second camera is at least partially compensated by transforming the second image data stream according to a set of transformation parameters; and the area (14) obscured for the first camera (2) is identified based on the first image data stream (15) and the transformed second image data stream.

3. Method according to claim 2, characterized in that the combined image data stream (16) is generated based on the first image data stream (15) and the transformed second image data stream, wherein the substitute image data (17) are generated based on the transformed second image data stream. A method according to claim 2 or 3, characterized in that at least one feature is identified that is represented by both the first image data stream (15) and the second image data stream; and the transformation parameter set is determined depending on a comparison of a representation of the at least one feature in the first image data stream (15) with a representation of the at least one feature in the second image data stream. A method according to any of the preceding claims, characterized in that the substitute image data (17) are superimposed on the first image data stream (15) to generate the combined image data stream (16).A method according to claim 5, characterized in that the substitute image data (17) are superimposed semi-transparently on the original image data of the first image data stream (15), which corresponds to the area (14) obscured for the first camera (2) in the first image data stream (15). A method according to any of the preceding claims, characterized in that the further vehicle (5) is located in the first field of view (8) and the area (14) obscured for the first camera (2) is obscured by the further vehicle (5) for the first camera (2). A method according to any of the preceding claims, characterized in that. The area (14) obscured by the first camera (2) is identified by means of at least one vehicle computing unit (10) of the ego vehicle (4); and / or the substitute image data (17) are determined by means of the at least one vehicle computing unit (10) of the ego vehicle (4); and / or the combined image data stream (16) is generated by means of the at least one vehicle computing unit (10) of the ego vehicle (4). A method according to one of the preceding claims, characterized in that the combined image data stream (16) is displayed on a vehicle-external display device (11) by means of a vehicle-external computer system (13).A method for remotely controlling an ego-vehicle (4), characterized in that a method for environmental perception according to claim 9 is performed; in response to the display of the combined image data stream (16) on the vehicle-external display device (11), a user input is detected by means of the vehicle-external computer system (13); and the ego-vehicle (4) is controlled at least partially automatically depending on the user input. A method according to claim 10, characterized in that a control command is transmitted to the ego-vehicle (4) by means of the vehicle-external computer system (13) depending on the user input; and the ego-vehicle (4) is controlled at least partially automatically depending on the control command.Environment detection system (1) comprising a first camera (2) for an ego vehicle (4) which is configured to generate a first image data stream (15), at least one communication interface for the ego vehicle (4) for wireless data transmission and at least one computing unit, characterized in that the at least one computing unit is configured to to receive a second image data stream generated by a second camera of another vehicle (5) via the at least one communication interface, wherein a first field of view (8) represented by the first image data stream (15) overlaps with a second field of view (9) represented by the second image data stream; to identify, based on the first image data stream (15) and the second image data stream, an area (14) in the first field of view (8) that is obscured for the first camera (2) but not obscured for the second camera; to generate, based on the second image data stream, substitute image data (17) that corresponds to the area (14) obscured for the first camera (2);and to generate a combined image data stream (16) based on the first image data stream (15), which represents the first field of view (8), and to display the substitute image data (17) in a region of the combined image data stream (16) that corresponds to the area (14) obscured for the first camera (2). Environment sensing system (1) according to claim 12, characterized in that the environment sensing system (1) has a vehicle-external display device (11) and the at least one computing unit includes a vehicle-external computer system (13), and the vehicle-external computer system (13) is configured to generate the combined image data stream (16) and to display it on the vehicle-external display device (11);or which includes at least one computing unit, at least one vehicle computing unit (10) configured to receive the second image data stream via the at least one communication interface, to identify the area (14) obscured for the first camera (2), to generate the substitute image data (17), to generate the combined image data stream (16), and to transmit the combined image data stream (16) via the at least one communication interface to the vehicle-external computer system (13), and the vehicle-external computer system (13) is configured to display the combined image data stream (16) on the vehicle-external display device (11). Vehicle guidance system for the teleoperated control of an ego-vehicle (4), characterized in that; The vehicle guidance system comprises an environment sensing system (1) according to claim 13; the vehicle-external computer system (13) is configured to detect user input in response to the display of the combined image data stream (16) on the vehicle-external display device (11); and the vehicle guidance system comprises an ego vehicle guidance system for the ego vehicle (4) that is configured to guide the ego vehicle (4) at least partially automatically depending on the user input.

15. Computer program product with commands which, when executed by an environment sensing system (1) according to claim 12 or 13, cause the environment sensing system (1) to perform a method according to any one of claims 1 to 9; and / or, when executed by a vehicle guidance system according to claim 14, cause the vehicle guidance system to perform a method according to claim 10 or 11.

Images