Ambient environment detection and remote operation guidance for own vehicle

By combining the camera image data stream of another vehicle with the image data stream of the guided vehicle, the problem of the vehicle's camera field of view is solved, and alternative image data is generated to supplement the occlusion area, improving the safety of remote operation guidance.

CN118946481APending Publication Date: 2024-11-12VALEO SCHALTER & SENSOREN GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380030523.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-28
Filing Date
2023-03-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the operation of highly automated vehicles, the field of view of the vehicle camera may be blocked by another vehicle driving in front, causing remote operators to be unable to obtain complete environmental information, posing safety risks.

Method used

By combining the camera image data stream of another vehicle with the image data stream of the guided vehicle, alternative image data is identified and generated to supplement the occlusion area, forming a combined image data stream, ensuring that the remote operator can obtain complete environmental information.

Benefits of technology

Improve the safety of remote operation guide bicycles, and enhance the decision-making ability of remote operators by providing information on the occlusion area to ensure that the vehicle can operate remotely more safely.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118946481A_ABST
    Figure CN118946481A_ABST
Patent Text Reader

Abstract

According to a method for detecting the surroundings, a first image data stream (15) is generated by means of a first camera (2) of an own vehicle (4) and a second image data stream generated by means of a second camera of another vehicle (5) is received, a field of view (8) represented by the first image data stream (15) overlapping a second field of view (9) represented by the second image data stream. In a first field of view (8) which is not occluded with respect to a second camera, a region (14) which is occluded with respect to the first camera (2) is identified. On the basis of the second image data stream, alternative image data (17) corresponding to an area (14) occluded with respect to the first camera (2) is generated. A combined image data stream representing the first field of view is generated on the basis of the first image data stream (15), in which alternative image data (17) is displayed in a region (14) of the combined image data stream (16) corresponding to a region (14) occluded with respect to the first camera (2).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for detecting a surrounding environment, wherein a first image data stream is generated by a first camera of an ego vehicle, and a method for remotely operating and guiding an ego vehicle. The present invention also relates to a corresponding surrounding environment detection system, a vehicle guidance system for remotely operating and guiding an ego vehicle, and a computer program product. Background Art

[0002] In the operation of highly automated vehicles (especially fully automated vehicles), if a situation arises that the vehicle cannot handle independently or cannot handle independently with sufficient safety, so-called teleoperation guidance can be used. In this case, a human teleoperator accesses data acquired from the vehicle's surroundings by the vehicle's various sensor systems (especially cameras) via a computer system external to the vehicle and can then remotely control the vehicle to resolve the situation.

[0003] However, in some cases, the field of view of the camera on the vehicle is partially blocked, such as by another vehicle driving ahead. In this case, the remote operator may not be able to consider the obscured information that is relevant to the remote control, which may pose a safety risk. Summary of the invention

[0004] This object is achieved by the respective subject matter of the independent claims. Advantageous developments and preferred embodiments are subject matter of the dependent claims.

[0005] The invention is based on the idea of ​​overcoming limitations in the field of view of a camera of a guided vehicle, which limitations are caused by occlusions such as by other vehicles travelling ahead, by combining the image data stream of a camera of another vehicle with the image data stream of the guided vehicle. In particular, substitute image data from the image data stream of the other vehicle are displayed in the occluded areas of the image data stream of the guided vehicle.

[0006] According to one aspect of the present invention, a method for detecting a surrounding environment is provided, wherein a first image data stream is generated by a first camera of the vehicle. A second image data stream generated by a second camera of another vehicle is received, in particular by at least one computing unit. The first field of view represented by the first image data stream overlaps with the second field of view represented by the second image data stream. Based on the first image data stream and the second image data stream, an area that is blocked relative to the first camera is identified in the first field of view, which area is not blocked relative to the second camera, in particular by the computing unit. Based on the second image data stream, substitute image data corresponding to the area blocked relative to the first camera is generated, in particular by at least one computing unit. Based on the first image data stream and in particular the second image data stream, a combined image data stream representing the first field of view is generated. The substitute image data is displayed in the area of ​​the combined image data stream corresponding to the area blocked relative to the first camera.

[0007] The second image data stream is received in particular by at least one computing unit, for example by at least one vehicle computing unit of the own vehicle, in particular by a first communication interface for wireless communication, for example for V2V or V2X communication. V2V stands for vehicle-to-vehicle, and V2X stands for vehicle-to-everything.

[0008] The at least one computing unit comprises, for example, at least one vehicle computing unit and can include, for example, a vehicle-external computer system, i.e., a computer system arranged outside the own vehicle and outside another vehicle. The vehicle-external computer system can be referred to as a server computer system or backend computer system. In particular, it can be a computer system for remotely controlling a vehicle, in particular the own vehicle.

[0009] The second image data stream is generated, in particular, by the second camera and transmitted from the other vehicle (for example, from another vehicle computing unit of the other vehicle, in particular via another communication interface for wireless communication (for example for V2V or V2X communication)) to the own vehicle (for example, in particular via the first communication interface to at least one vehicle computing unit).

[0010] For example, the image data stream may represent a sequence of two or more consecutive images. The first image data stream (in particular the corresponding first image of the first image data stream) represents in particular the external surroundings of the own vehicle. It can also be understood that the first field of view camera of the field of view of the first camera is therefore at least partially located in the external surroundings of the own vehicle. The second image data stream (in particular the corresponding second image of the first image data stream) represents in particular the external surroundings of another vehicle. It can also be understood that the second field of view camera of the field of view of the second camera is therefore at least partially located in the external surroundings of the other vehicle.

[0011] Since the first field of view and the second field of view overlap, the first and second image data streams generally represent partially matching content. However, since the area that is occluded for the first camera is not occluded for the second camera, the second image data stream represents content that is not represented by the first image stream due to occlusion relative to the first camera. For example, the occlusion may be caused by an occluding object in the first field of view. The occluding object may be, for example, another vehicle itself, but not necessarily the other vehicle.

[0012] Thus, the replacement image data may be generated, for example, based on image data of the second image data stream, which image data represent an area of ​​the second field of view which is also located in the first field of view and which would be represented by an area of ​​the first image data stream if no occluding object was present. The corresponding image data of the second image data stream may in particular be processed or modified to generate the replacement image data, in particular to adapt the representation to the first image data stream.

[0013] The combined image data stream corresponds in particular to a combination of the first image data stream or an edited variant of the first image data stream with substitute image data.

[0014] Outside the area corresponding to the area obscured relative to the first camera, for example, the combined image data stream may be identical to the first image data stream. In the area corresponding to the area obscured relative to the first camera, the combined image data stream may correspond to, for example, substitute image data or to a superposition of substitute image data with the first image data stream. There may also be transition areas between these areas.

[0015] Therefore, due to the replacement image data, the combined image data stream contains information that would otherwise not be available to the ego vehicle due to occlusion. As a result, an observer of the combined image data stream (e.g. a teleoperator who is remotely controlling the ego vehicle via teleoperation guidance) can learn more about the current state of the ego vehicle's surroundings from the ego vehicle, which ultimately improves the safety of the ego vehicle when operating via teleoperation guidance.

[0016] According to at least one embodiment of the method for detecting surroundings according to the invention, by transforming the second image data stream according to, for example, a predefined set of transformation parameters, deviations of one or more camera extrinsic parameters and / or one or more camera intrinsic parameters between the first camera and the second camera are at least partially compensated, for example by at least one computing unit. Areas that are obscured relative to the first camera are identified based on the first image data stream and the transformed second image data stream, in particular by at least one computing unit, for example by at least one vehicle computing unit identifying the camera.

[0017] The camera external parameters include, for example, the attitude, i.e., the position and orientation, of the camera coordinate system of the corresponding camera relative to the reference coordinate system. The camera coordinate system is rigidly connected to the corresponding camera. For example, the reference coordinate system can be rigidly connected to the corresponding vehicle or the surrounding environment of the corresponding vehicle. A common reference coordinate system can also be selected for cameras of different vehicles. For example, the attitude can be defined by six parameters, such as three position parameters and three orientation parameters. The three position parameters can be, for example, the three-dimensional coordinates of the coordinate origin of the camera coordinate system in the reference coordinate system. For example, the three orientation parameters can be three orientation angles, such as Euler angles, which can also be called pitch angle, yaw angle and roll angle.

[0018] Camera intrinsic parameters may include parameters that define how the camera images points near the camera, in particular on the camera's image sensor. Thus, camera intrinsic parameters may describe or define, for example, the imaging function of the camera lens, the focal length, etc. Camera intrinsic parameters may also include the aspect ratio, resolution and / or pixel density of the image sensor. Camera intrinsic parameters may also relate to parameters related to camera internal pre-processing of the acquired image data, such as parameters related to white balance, image sharpening, color correction, etc.

[0019] According to at least one embodiment, a combined image data stream is generated based on the first image data stream and the transformed second image data stream, wherein the substitute image data are generated based on the transformed second image data stream, in particular by at least one computing unit, for example by at least one vehicle computing unit.

[0020] The at least partial compensation of the deviation of one of the camera extrinsic parameters and / or camera intrinsic parameters allows, for example, deviations in the scale and / or scaling of image objects between the first image data stream and the second image data stream to be at least partially compensated, an offset and / or displacement of the first field of view relative to the second field of view to be at least partially compensated, and / or perspective deviations between the first image data stream and the second image data stream to be at least partially compensated. This ensures that the replacement image data in combination with the image data from the first image data stream allows a consistent representation of the surroundings, which ultimately improves safety in the application case of a teleoperated guided ego vehicle.

[0021] According to at least one embodiment, at least one feature represented by the first image data stream and the second image data stream is identified, in particular by at least one computing unit, such as at least one vehicle computing unit, in particular based on the first image data stream and the second image data stream. A set of transformation parameters is determined, in particular by at least one computing unit, such as at least one vehicle computing unit, based on a comparison of the representation of the at least one feature in the first image data stream with the representation of the at least one feature in the second image data stream.

[0022] By identifying and comparing given features, the influence of deviations of one or more camera extrinsic parameters and / or one or more camera intrinsic parameters between the first camera and the second camera can be extracted directly from the image data stream. Thus, advantageously, deviations of camera parameters that may not have a relevant influence are not explicitly taken into account and therefore do not need to be explicitly determined. This allows for a more efficient compensation of deviations of camera parameters.

[0023] According to at least one embodiment, the tonal value distribution and / or color value distribution of the second image data stream is compared with the tonal value distribution and / or color value distribution of the first image data stream, in particular by at least one computing unit, for example by at least one vehicle computing unit. Based on the result of the comparison, the converted second image data stream is corrected or the second image data stream is corrected before the conversion. Substitute image data are generated based on the converted and corrected second image data stream.

[0024] In particular, this ensures that brightness and white balance are consistent in the combined image data stream, which ultimately further increases safety in application cases where teleoperated self-driving vehicles are being guided.

[0025] The tonal value distribution and / or the color value distribution can be given, for example, by one or more corresponding histograms.

[0026] According to at least one embodiment, substitute image data are superimposed on the first image data stream, in particular on areas of the first image data stream corresponding to areas occluded relative to the first camera, in order to generate a combined image data stream.

[0027] For example, the substitute image data is superimposed in a partially transparent manner on original image data of the first image data stream, the original image data corresponding to areas of the first image data stream that are occluded relative to the first camera.

[0028] In the combined image data stream, both the surrogate image data and the original image data are represented. Thus, the combined image data stream represents both the occluding object and what is behind the occluding object when viewed from the first camera. As a result, a viewer (e.g., a remote operator) can consider more information when making decisions, which ultimately further improves safety in autonomous vehicle remote operation guidance applications.

[0029] According to at least one embodiment, the other vehicle is located in the first field of view, in particular the other vehicle is represented by the first image data stream, and the area obscured relative to the first camera is obscured by the other vehicle from the first camera. In other words, the other vehicle is an obstructing object.

[0030] For example, another vehicle may be driving in front of the ego vehicle on the road, in particular in the same lane as the ego vehicle or in another lane, such as an adjacent lane. For example, the first camera may be a forward-facing camera of the ego vehicle, also referred to as a front camera. The second camera may be, for example, a front camera of the other vehicle.

[0031] According to at least one embodiment, the area that is blocked relative to the first camera is detected by at least one vehicle computing unit of the own vehicle.

[0032] Alternatively, the at least one vehicle computing unit can transmit the first image data stream and the second image data stream to an external computer system, in particular via the first communication interface or the second communication interface, and the external computer system can identify the obscured area relative to the first camera.

[0033] According to at least one embodiment, the substitute image data are determined by at least one vehicle computing unit of the own vehicle.

[0034] Alternatively, the at least one vehicle computing unit can transmit the second image data stream and optionally the first image data stream to an external computer system, in particular via the first communication interface or the second communication interface, and the external computer system can determine the replacement image data.

[0035] According to at least one embodiment, the combined image data stream is generated by at least one vehicle computing unit of the ego vehicle.

[0036] Alternatively, the at least one vehicle computing unit can transmit the first image data stream to the external computer system, in particular via the first communication interface or the second communication interface. Thus, the at least one vehicle computing unit can also transmit the second image data stream or the replacement image data to the external computer system. The external computer system can then generate a combined image data stream.

[0037] In a preferred embodiment, the steps of identifying the occluded area, generating the substitute image data and generating the combined image data stream are performed by at least one vehicle computing unit. For example, the combined image data stream is transmitted to the external computer system by at least one vehicle computing unit, in particular via the first communication interface or the second communication interface.

[0038] According to at least one embodiment, the combined image data stream is displayed by a vehicle-external computer system on a vehicle-external display device, such as one or more screens or monitors, or a visual output device worn on the head, also referred to as an HMD (“head mounted display”). Beforehand, the combined image data stream is generated, for example, by at least one vehicle computing unit and transmitted to the external computer system, in particular via the first communication interface or the second communication interface, or the combined image data stream is generated by the external computer system.

[0039] According to another aspect of the invention, a method for remotely operating and guiding an ego vehicle is proposed. To this end, the method according to the invention for detecting the surrounding environment is implemented, in particular by displaying a combined image data stream on a vehicle-external display device by a vehicle-external computer system. In response to displaying the combined image data stream on the vehicle-external display device, in particular after or simultaneously with the display, a user input, in particular an input of a remote operator, is captured by the vehicle-external computer system. Based on the user input, the ego vehicle is at least partially automatically guided, in particular the ego vehicle is remotely controlled.

[0040] According to at least one embodiment of the method for remotely operating the guidance of an ego vehicle, a vehicle-external computer system is used to generate control commands based on user input and transmit them to the ego vehicle. Based on the control commands, the ego vehicle is guided at least partially automatically, in particular by an ego vehicle guidance system of the ego vehicle.

[0041] Here, an ego vehicle guidance system may be understood in particular as an electronic system which is configured to guide the ego vehicle fully automatically or fully autonomously, in particular without control intervention by the driver. The ego vehicle automatically performs all necessary functions, such as steering, braking and / or acceleration operations, observation and detection of road traffic and appropriate reactions. In particular, the ego vehicle guidance system may enable a fully automatic or fully autonomous driving mode 5 of a motor vehicle of level 5 according to the SAE J3016 classification. In particular, the ego vehicle guidance system may enable a fully automatic or fully autonomous driving mode of a motor vehicle of level 4 according to the SAE J3016 classification. "SAE J3016" here and below refers to the corresponding standard in its June 2018 version.

[0042] The ego vehicle guidance system may generate one or more control signals, for example depending on the control command, and transmit them to one or more actuators of the ego vehicle so that they influence or perform lateral and / or longitudinal control of the ego vehicle according to the control command.

[0043] According to another aspect of the present invention, a surroundings detection system is provided, which includes a first camera for a self-vehicle, which is configured to generate a first image data stream. The surroundings detection system has at least one communication interface for the self-vehicle for wireless data transmission, and at least one computing unit, in particular at least one vehicle computing unit. At least one computing unit, in particular at least one vehicle computing unit, is configured to receive a second image data stream generated by a second camera of another vehicle through at least one communication interface, such as a first communication interface, wherein a first field of view represented by the first image data stream overlaps with a second field of view represented by the second image data stream. At least one computing unit, in particular at least one vehicle computing unit, is configured to identify an area in the first field of view that is blocked relative to the first camera based on the first image data stream and the second image data stream, while the area is not blocked for the second camera, and to generate substitute image data corresponding to the area blocked relative to the first camera based on the second image data stream. At least one computing unit, in particular at least one vehicle computing unit, is configured to generate a combined image data stream representing the first field of view based on the first image data stream and in particular based on the second image data stream, and to display substitute image data in an area corresponding to the area blocked relative to the first camera in the combined image data stream.

[0044] According to at least one embodiment of the surroundings detection system, the surroundings detection system is designed as an ego vehicle guidance system for an ego vehicle.

[0045] According to at least one embodiment of the surroundings detection system, it comprises a vehicle-external display device, and the at least one computing unit comprises a vehicle-external computer system.

[0046] In such an embodiment, the surrounding environment detection system may also be referred to as a vehicle guidance system for remote operation guidance of the own vehicle.

[0047] According to at least one embodiment of the surrounding environment detection system, the vehicle-external computer system is configured to generate a combined image data stream and display it on a vehicle-external display device. In this case, the vehicle-external computer system can, for example, receive the first image data stream and the second image data stream from at least one vehicle computing unit.

[0048] According to at least one embodiment of the surroundings detection system, at least one vehicle computing unit is configured to receive the second image data stream via at least one communication interface, in particular the first communication interface, to identify the area obscured relative to the first camera, to generate alternative image data, to generate a combined image data stream, and to transmit the combined image data stream via at least one communication interface, in particular the first communication interface or the second communication interface, to a vehicle-external computing system. The vehicle-external computing system is configured to display the combined image data stream on a vehicle-external display device.

[0049] According to at least one embodiment of the vehicle guidance system, for remote controlled guidance of the ego vehicle, the vehicle-external computer system is configured to detect a user input in response to display of the combined image data stream on a vehicle-external display device. For remote controlled guidance, the vehicle guidance system has an ego vehicle guidance system for the ego vehicle, which is configured to guide the ego vehicle at least partially automatically based on the user input.

[0050] According to at least one embodiment of the vehicle guidance system, for remote controlled guidance of the ego vehicle, the vehicle-external computer system is configured to generate control commands based on user input and transmit them to the ego vehicle. The ego vehicle guidance system is configured to guide the ego vehicle at least partially automatically based on the control commands.

[0051] Further embodiments of the surroundings detection system of the invention and the vehicle guidance system of the invention are directly derived from the various configurations of the method of the invention for detecting the surroundings and the method of the invention for remote control guidance of an own vehicle, and vice versa. In particular, the individual features and the corresponding descriptions of the various embodiments of the method of the invention can be transferred analogously to the corresponding embodiments of the surroundings detection system of the invention and the vehicle guidance system of the invention. In particular, the surroundings detection system of the invention is designed or programmed to perform the method of the invention for detecting the surroundings. In particular, the surroundings detection system of the invention performs the method of the invention for detecting the surroundings. In particular, the vehicle guidance system of the invention is designed or programmed to perform the method of the invention for remote control guidance of an own vehicle. In particular, the vehicle guidance system of the invention performs the method of the invention for remote control guidance of an own vehicle.

[0052] According to another aspect of the present invention, a first computer program having first instructions is provided. When the first instructions are executed by the surrounding environment detection system according to the present invention, the first instructions cause the surrounding environment detection system to execute the method for detecting surrounding environment according to the present invention.

[0053] According to another aspect of the present invention, a second computer program having second instructions is provided. When the second instructions are executed by a vehicle guidance system for remote operation guidance of an ego vehicle according to the present invention, the second instructions cause the vehicle guidance system to execute the method for remote operation guidance of an ego vehicle according to the present invention.

[0054] According to a further aspect of the invention, a computer-readable storage medium is provided which stores a first computer program according to the invention and / or a second computer program according to the invention.

[0055] The first computer program, the second computer program and the computer-readable storage medium may each be regarded as a respective computer program product having first instructions and / or second instructions.

[0056] The term computing unit may be understood in particular as a data processing device comprising a processing circuit. Thus, the computing unit may process data, in particular for performing computing operations. Optionally, these operations also include operations for performing indicated accesses to data structures, such as look-up tables (LUTs).

[0057] In particular, the computing unit may comprise one or more computers, one or more microcontrollers and / or one or more integrated circuits, such as one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs) and / or one or more systems on a chip (SoCs). The computing unit may also comprise one or more processors, such as one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs) and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit may also comprise a physical or virtual group of computers or other types of the above-mentioned units.

[0058] In various exemplary embodiments, a computing unit includes one or more hardware and / or software interfaces and / or one or more storage units.

[0059] The memory cell may be implemented as a volatile data memory, such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), or as a non-volatile data memory, such as a read-only memory (ROM), a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory or a flash EEPROM, a ferroelectric random access memory (FRAM), a magnetoresistive random access memory (MRAM), or a phase change random access memory (PCRAM).

[0060] If within the scope of the present disclosure it is mentioned that a component of the surroundings detection system or the vehicle guidance system according to the invention, in particular at least one computing unit, at least one vehicle computing unit and / or a vehicle-external computer system, is configured, embodied, designed or otherwise performs or implements a specific function, achieves a specific effect or serves a specific purpose, this is to be understood to mean that the component, through appropriate adaptation, appropriate programming, appropriate physical design, etc., is specifically and actually able to perform or implement this function, achieve this effect or serve this purpose, which goes beyond the basic or theoretical usability or suitability of the component for this function, effect or purpose.

[0061] Further features of the invention can be found in the claims, the drawings and the description of the drawings. The features and feature combinations mentioned above in the description of the drawings and the features and feature combinations described below in the description of the drawings and the features and feature combinations shown in the drawings may be included in the invention not only in the combination specified in each case but also in other combinations. In particular, the invention may also include embodiments and feature combinations which do not have all the features of the original claims. Furthermore, the invention may also include embodiments and feature combinations which exceed or differ from the feature combinations listed later in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be explained in more detail below based on specific exemplary embodiments and with reference to relevant schematic diagrams. In the figures, identical or functionally identical elements may have identical reference symbols. For different figures, the description of identical or functionally identical elements may not necessarily be repeated.

[0063] In the figure:

[0064] Figure 1 A schematic diagram showing an exemplary embodiment of a surrounding environment detection system according to the present invention; and

[0065] Figure 2 A schematic flow chart of an exemplary embodiment of a method for detecting surroundings according to the present invention is shown. DETAILED DESCRIPTION

[0066] Figure 1 Schematically shows an exemplary embodiment of the surrounding environment detection system 1 according to the present invention, which also particularly shows a vehicle guidance system for remotely operating and guiding a self-vehicle 4 according to the present invention.

[0067] The own vehicle 4 is traveling on the road behind the other vehicle 5. For example, the other vehicles 6 and 7 may be traveling in front of the other vehicle 5.

[0068] The surroundings detection system 1 has a first camera 2 of an own vehicle 4, in particular a front camera, which is configured to generate a first image data stream 15, which represents a first field of view 8 of the first camera 2. Another vehicle 5 and, for example, another vehicle 6, 7 are located in the first field of view 8, wherein the other vehicle 5 blocks an area 14 in the first field of view 8 of the first camera 2, so that in particular the other vehicle 6, 7 is partially blocked in the first image data stream 15.

[0069] The other vehicle 5 has a second camera 3, in particular a front camera, which is configured to generate a second image data stream, which represents a second field of view 9 of the second camera 3. The other vehicles 6, 7 are located, for example, in the second field of view 9 and are in particular in this case not obstructed relative to the second camera 3, or less obstructed than relative to the first camera 2.

[0070] The own vehicle 4 and the other vehicles 5 each have a communication interface for wireless data transmission, such as a V2V or V2X interface. In addition, the surroundings detection system 1 has a vehicle computing unit 10 and a vehicle-external computer system 10 , which is, for example, part of a backend for remotely operating vehicle guidance.

[0071] The vehicle computing unit 10 receives the second image data stream from the other vehicle 5 (for example from another vehicle control unit of the other vehicle 5 ) via the communication interface.

[0072] The vehicle computing unit 10 identifies the obscured area 14 relative to the first camera 2, for example based on a comparison of the first image data stream 15 with the second image data stream, and generates substitute image data 17 corresponding to the obscured area 14 relative to the first camera 2 based on the second image data stream. The vehicle computing unit 10 then generates a combined image data stream 16 representing the first field of view 8, wherein the substitute image data 17 is displayed in the area of ​​the combined image data stream 16 corresponding to the obscured area 14 relative to the first camera 2, for example superimposed in a semi-transparent manner on the original image data of the first image data stream 15.

[0073] The vehicle computing unit 10 can wirelessly transmit the combined image data stream 16 to the vehicle-external computer system 13. In particular, the rear-end surroundings detection system 1 has a vehicle-external display device 11. The vehicle-external computer system 13 is configured to generate the combined image data stream 16 and display it on the vehicle-external display device 11.

[0074] The remote operator 12 may analyze the displayed combined image data stream 16 and, in response thereto, perform user inputs on an input device of the vehicle-external computer system 13. Based on the user inputs, the vehicle-external computer system 13 may transmit control commands to the vehicle computing unit 10. Based on the control commands, the ego vehicle 4 may be guided at least partially automatically.

[0075] Figure 2 A schematic flow chart of an exemplary embodiment of a method for detecting surroundings according to the present invention is shown.

[0076] In step S0a, another vehicle 5 is identified and selected, for example based on its V2V or V2X functionality and / or relative position to the ego vehicle 4. Furthermore, a second image data stream is generated and transmitted to the ego vehicle 4. In step S0b, a first image data stream is generated.

[0077] In step S1a, features are identified in the second image data stream and in step S1b, features are identified in the first image data stream 15. In step S2, matching features are then identified in the first image data stream 15 and the second image data stream based on the previously identified features. These features may be, for example, objects or edges in the respective image data streams. In step S3, the matching features are compared and in step S4, a transformation parameter set is determined based on the comparison, which transformation parameter set approximately transforms the matching features of the second image data stream into the matching features of the first image data stream 15. In particular, the transformation parameters of the transformation parameter set are optimized to achieve an optimal superposition of the matching features. The transformation parameter set particularly relates to scale, displacement and / or perspective adjustments.

[0078] Since the first camera 2 and the second camera 3 do not necessarily have the same white balance, brightness and contrast, the color of the second image data stream can also be selectively adjusted. For example, this can be achieved by adjusting the histogram of the first image data stream 15 and the second image data stream.

[0079] In step S6, the obscured areas 14 are masked. To this end, the relevant areas are identified by comparing the two image data streams and detecting areas that are significantly different. These areas are likely to be obscured by another vehicle 5. Alternatively, if available, the position data of the vehicle traveling ahead can be used. In step S7, a combined image data stream 16 is generated.

[0080] In known remote controlled vehicles, the remote operator's field of view is limited to a camera image of the vehicle to be guided. In the absence of direct feedback from the vehicle and the surrounding environment, it is desirable to assist the remote operator as much as possible.

[0081] The present invention, in various embodiments, enables reducing the teleoperator's visual limitations due to obstructing objects by using camera data of other road users.

[0082] In various embodiments of the present invention, V2X communication technology is used, which allows communication between individual vehicles and fixed installed units to exchange sensor data.

Claims

1. A method for detecting a surrounding environment, wherein a first image data stream (15) is generated by a first camera (2) of an ego vehicle (4), Features - receiving a second image data stream generated by a second camera of another vehicle (5), wherein a first field of view (8) represented by the first image data stream (15) overlaps a second field of view (9) represented by the second image data stream; - based on the first image data stream (15) and the second image data stream, identifying in the first field of view (8) an area (14) that is obscured relative to the first camera (2) and that is not obscured relative to the second camera; - generating, based on the second image data stream, substitute image data (17) corresponding to the area (14) obscured relative to the first camera (2); and - Based on the first image data stream (15), generating a combined image data stream representing the first field of view, wherein substitute image data (17) are displayed in an area of ​​the combined image data stream (16) corresponding to the area (14) that is obscured relative to the first camera (2).

2. The method according to claim 1, Features - at least partially compensating for a deviation in one or more camera extrinsic parameters and / or one or more camera intrinsic parameters between the first camera (2) and the second camera by transforming the second image data stream according to the set of transformation parameters; and - identifying an area (14) that is obscured relative to the first camera (2) based on the first image data stream (15) and the transformed second image data stream.

3. The method according to claim 2, Features A combined image data stream (16) is generated based on the first image data stream (15) and the transformed second image data stream, wherein substitute image data (17) are generated based on the transformed second image data stream.

4. The method according to claim 2 or 3, Features - identifying at least one feature represented by both the first image data stream (15) and the second image data stream; and - determining a set of transformation parameters based on a comparison of a representation of at least one feature in the first image data stream (15) and a representation of at least one feature in the second image data stream.

5. A method according to any one of the preceding claims, It is characterized in that The substitute image data (17) is superimposed on the first image data stream (15) to generate a combined image data stream (16).

6. The method according to claim 5, It is characterized in that The substitute image data (17) are superimposed in a partially transparent manner on the original image data of the first image data stream (15), the original image data corresponding to the area (14) that is obscured in the first image data stream (15) relative to the first camera (2).

7. A method according to any one of the preceding claims, It is characterized in that The other vehicle (5) is located in the first field of view (8), and the area (14) that is obscured relative to the first camera (2) is obscured by the other vehicle (5) relative to the first camera (2).

8. The method according to any one of the preceding claims, Features - identifying an area (14) that is blocked relative to the first camera (2) by at least one vehicle computing unit (10) of the own vehicle (4); and / or - determining the substitute image data (17) by at least one vehicle computing unit (10) of the own vehicle (4); and / or - generating a combined image data stream (16) by at least one vehicle computing unit (10) of the own vehicle (4).

9. The method according to any one of the preceding claims, Features The combined image data stream (16) is displayed on a vehicle-external display device (11) via a vehicle-external computer system (13).

10. A method for remotely controlling a vehicle (4), Features - Execute the method for detecting the surrounding environment as claimed in claim 9; - capturing user input via the vehicle external computer system (13) in response to display of the combined image data stream (16) on the vehicle external display device (11); and - Controlling the vehicle (4) at least partially automatically based on user input.

11. The method according to claim 10, Features - based on user input, the vehicle external computer system (13) is used to send control commands to the vehicle (4); and - guiding the ego vehicle (4) at least partially automatically according to the control command.

12. A surroundings detection system (1), comprising a first camera (2) for a vehicle (4), at least one communication interface for the vehicle (4), and at least one computing unit, wherein the first camera (2) is configured to generate a first image data stream (15), and the at least one communication interface is used for wireless data transmission. Features At least one computing unit is configured to - receiving, via at least one communication interface, a second image data stream generated by a second camera of another vehicle (5), wherein a first field of view (8) represented by the first image data stream (15) overlaps a second field of view (9) represented by the second image data stream; - based on the first image data stream (15) and the second image data stream, identifying an area (14) in the first field of view (8) that is obscured for the first camera (2) and that is not obscured for the second camera; - generating, based on the second image data stream, substitute image data (17) corresponding to the area (14) obscured by the first camera (2); and - Based on the first image data stream (15), a combined image data stream (16) representing the first field of view (8) is generated, while displaying substitute image data (17) in an area of ​​the combined image data stream (16) corresponding to the area (14) that is obscured relative to the first camera (2).

13. The surrounding environment detection system (1) according to claim 12, Features The surroundings detection system (1) comprises a vehicle-external display device (11), and at least one computing unit comprises a vehicle-external computer system (13), and - the vehicle-external computer system (13) is configured to generate a combined image data stream (16) and display it on a vehicle-external display device (11); or - the at least one computing unit comprises at least one vehicle computing unit (10) configured to receive the second image data stream via the at least one communication interface, identify an obstructed area (14) relative to the first camera (2), generate substitute image data (17), generate a combined image data stream (16) and transmit the combined image data stream (16) to the vehicle-external computer system (13) via the at least one communication interface, and the vehicle-external computer system (13) is configured to display the combined image data stream (16) on a vehicle-external display device (11).

14. A vehicle guidance system for remotely controlling a vehicle (4), Features - A vehicle guidance system having a surroundings detection system (1) according to claim 13; - the vehicle-external computer system (13) is configured to capture user input responsive to display of the combined image data stream (16) on the vehicle-external display device (11); and The vehicle guidance system has an ego vehicle guidance system for an ego vehicle (4), which is configured to guide the ego vehicle (4) at least partially automatically based on user input.

15. A computer program product having instructions, the instructions - when executed by the surrounding environment detection system (1) according to any one of claims 12 or 13, causing the surrounding environment detection system (1) to execute the method according to any one of claims 1 to 9; and / or - when executed by a vehicle guidance system according to claim 14, causes the vehicle guidance system to perform a method according to any one of claims 10 or 11.