Vehicle safety and / or driver assistance method, arrangement and data processing program product using a connectable image acquisition system for road vehicles

Abstract

Vehicle safety and / or driver assistance method, arrangement, and data processing program product using a first image acquisition system with a first field of view suitable for acquiring speed data, generating first field of view depth data and / or object detection distance data, in particular using an artificial intelligence algorithm, evaluating, based on the acquired speed data, whether a minimum field of view depth criterion and / or whether a safety criterion of the minimum object distance is met if the minimum field of view depth criterion and / or the safety criterion of the minimum object distance is not met, combining the image data of the first image acquisition system with the image data of the second image acquisition system to generate a data structure from combined image data, evaluating on the basis of the combined image data,whether a minimum field of view depth criterion and / or a minimum object distance safety criterion is met, and depending on the evaluation of whether the criteria are met or not met, outputting and / or displaying safety data, wherein the safety data indicates whether the performance of a particular vehicle maneuver is safe or unsafe.

Description

FIELD OF THE INVENTION

[0001] The present invention relates to a method, an arrangement and a data processing program product for vehicle safety and / or driver assistance using connectable image capture systems for road vehicles, in particular using connectable rearview systems, in particular using connectable camera-based rearview systems, in particular for virtually expanding the field of view of such image capture systems for object detection, in particular when the field of view of at least one image capture system is restricted. TECHNICAL BACKGROUND

[0002] US patent US 10,501,015 B2 discloses a method and apparatus for augmented view, the method comprising generating a virtual view image of a vehicle based on a surrounding view of the vehicle, generating an augmented virtual view image based on the virtual view image of the vehicle and a received virtual view image of another vehicle, and displaying the augmented virtual view image to a user.An embodiment in this document provides a surroundings viewing method, the method comprising: generating, by a vehicle, a first image of the vehicle based on a surroundings image of the vehicle; receiving a second image generated by another vehicle on a road; determining a location relationship between the vehicle and the other vehicle; generating an augmented image using the first image and the second image; and outputting the generated augmented image, wherein generating the augmented image comprises selecting an overlapping area based on the location relationship that appears in both the first image and the second image, and generating the augmented image to include the overlapping area.Another embodiment in this document provides a surroundings viewing device, the device comprising: an image generator configured to generate a first image of a vehicle based on a surroundings image of the vehicle; a receiver configured to receive a second image generated by another vehicle on a road; and an augmented image generator configured to determine a location relationship between the vehicle and the other vehicle; generate an augmented image using the first image and the second image, select an overlapping area appearing in both the first image and the second image based on the location relationship, and generate the augmented image to include the overlapping area; and a display configured to display the generated augmented image.

[0003] Vehicles, especially road vehicles, must be equipped with exterior rearview systems that allow the driver to observe the area surrounding the vehicle beyond the driver's peripheral vision. To allow the driver to observe the area surrounding the vehicle, exterior rearview systems are mounted on the exterior of road vehicles. The area surrounding the vehicle includes the rear of the vehicle and the left and right sides of the vehicle. In addition, some exterior rearview systems are configured to effectively cover the driver's blind spot.

[0004] Exterior rearview systems typically utilize some type of reflective surface, particularly a mirror, or a camera system. Furthermore, exterior rearview systems, particularly systems housing a camera system, are configured to be mounted on the vehicle body and incorporate a variety of connectors, sensors, and / or actuators to ensure proper functionality.

[0005] A rear-view device for a motor vehicle provides an image of the rear of the motor vehicle that meets at least the legal requirements and belongs to a subgroup of devices for indirect vision. These provide images and views of objects that are not in the driver's direct field of vision, i.e. in the directions opposite, to the left, to the right, below and / or above the driver's line of sight. It may happen that the driver's view is not entirely satisfactory, particularly in the direction of vision. For example, visibility may be obstructed by parts of the vehicle itself, such as parts of the car, in particular the A-pillar, the roof structure and / or the bonnet, and visibility may be obstructed by other vehicles and / or objects external to the vehicle, which may reduce visibility to such an extent that the driver is unable to assess a driving situation or can only assess it in an incompletely satisfactory manner.In addition, the driver may not be able to perceive the situation in or out of line of sight as required to control the vehicle appropriately. Therefore, a rearview device can also be designed to process the information according to the driver's capabilities to provide the best possible understanding of the situation.

[0006] Various functions and devices can be incorporated into and / or controlled by rearview devices, in particular those containing cameras. Particularly useful are functions and devices for improving, extending and / or maintaining the functionality of the rearview device under normal or extreme conditions. These may include heating and / or cooling devices, cleaning devices such as windshield wipers, actuators for moving the rearview device or parts thereof, such as a display, a camera system and / or parts of a camera system, e.g., lenses, filters, light sources, adaptive optics such as deformable mirrors, sensors and / or mirrors, and / or actuators for triggering movements of other objects, e.g., parts of the vehicle and / or objects in the vehicle's surroundings.

[0007] The camera may, for example, contain one or more CCD or CMOS sensors. The housing of a camera module may be made of plastic, metal, glass, another suitable material, and / or any combination thereof, and may be used in combination with the techniques described below to alter or modify the properties of the material or its surface. Various types of fastening devices may be used to attach the camera module to the vehicle or other components, such as positive interlocking fasteners and / or non-positive interlocking fasteners.

[0008] A person skilled in the art can find further details and indications for implementing one or more of the aforementioned technical features, for example, in the earlier patent application US 11,014,490 B2, which was filed by the same patent applicant as the applicant for this patent application and which is hereby incorporated by reference into the disclosure of this patent application. The individual features of this earlier patent application, which was filed by the applicant for this patent application, are hereby incorporated by reference into the description of this invention where and to the extent that it may be useful for the person skilled in the art that such individual features of this earlier patent application be combined with the features of the invention disclosed here and / or used to carry out the invention disclosed here.

[0009] In the earlier patent application US 2019 / 318178 A1, filed by the same patent applicant as the applicant of this patent application, and hereby incorporated by reference into the disclosure of this patent application, the person skilled in the art will find further details and indications on how to implement a method for obtaining 3D information from objects depicted in at least two images, as well as a device for performing the respective steps of the method and a system including such a device for implementation in a vehicle including such a device or system. In particular, in connection with the aforementioned document and in connection with the invention, images can be captured by at least two on-board image sensors.

[0010] In cases where some or more image sensors are unavailable, the invention can apply an artificial intelligence (AI) algorithm to detect objects and / or distances from 2D images or video sequences. Deep learning models, such as convolutional neural networks (CNNs), can be trained to detect and estimate distances to objects based on visual features such as size, shape, perspective, and / or texture. Such deep learning models can be designed to automatically learn hierarchical representations of features in images. In a training phase, the deep learning model is presented with labeled datasets. The model learns to detect patterns, textures, shapes, and features in images that are relevant to the task. It adjusts its internal parameters (weights and biases) through a process called backpropagation and optimization techniques such as gradient descent.An alternative approach is learning from unlabeled data, such as zero-shot learning AI algorithms. An AI algorithm using zero-shot learning can be used for object and distance detection by using a pre-trained language model and a neural network architecture to generate descriptions of objects and estimate their distances. The pre-trained language model can automatically generate descriptions of objects based on their visual features, such as size, shape, perspective, and / or texture. The neural network can then use these descriptions to detect and classify objects and determine distances to objects. The pre-trained language model can generate descriptions of distances to objects, and the neural network can use these descriptions to estimate the distance to the detected objects.This means that AI models trained on labeled datasets and / or zero-shot learning can be used for object and distance detection to create descriptions of objects and estimate their distances.

[0011] An artificial intelligence algorithm applicable for use within the scope of the invention may be configured to comprise a generative pre-trained transformer that uses specific algorithms to find patterns in data sequences, in particular in image data representing images acquired by one or more on-board image acquisition systems and / or in image data representing sequences of images acquired by one or more on-board image acquisition systems. Therefore, such an AI algorithm that can be used within the scope of the invention may comprise a machine learning model in the form of a neural network, in particular in the form of a transversal neural network and / or in the form of a convolutional neural network, in correlation with a generative pre-trained transformer.

[0012] An artificial intelligence algorithm that can be used in the context of the invention can be designed to include a generative transverse neural network (TNN) algorithm, module, or method. The transverse neural network (TNN) algorithm, module, or method can be used to determine the distance of an object in a camera image. Transverse neural networks are special neural network architectures developed to model spatial information and solve tasks such as distance determination or depth perception. Those skilled in the art can find further details and guidance on implementing a generative transverse neural network (TNN) algorithm, module, or method in the following prior art document, which is hereby incorporated by reference into the disclosure of this patent application: Koch, T., Liebel, L., Fraundorfer, F., Körner, M. (2019). Evaluation of CNN-Based Single-Image Depth Estimation Methods. In: Leal-Taixé, L., Roth, S. (eds) Computer Vision - ECCV 2018 Workshops. ECCV 2018. Lecture Notes in Computer Science(), vol 11131. Springer, Cham., https: / / doi.org / 10.1007 / 978-3-030-11015-4_25

[0013] There are various approaches to using a transversal neural network (TNN) algorithm, module, or method for distance estimation. A common method is the use of so-called depth estimation networks based on a transversal neural network (TNN) algorithm, module, or method. These networks are trained with large amounts of coupled images and depth information to capture the spatial relationships between pixels.

[0014] The depth estimation network takes a camera image as input and generates an image corresponding to the depth information as output. By analyzing the spatial features and patterns in the images, they can estimate distance data—that is, how far the objects in the image are from the image capture device that captured the output image.

[0015] The person skilled in the art may use any of the aforementioned artificial intelligence algorithms, modules or methods, or parts thereof, or a combination thereof within the scope of the invention described in this document. SUMMARY OF THE INVENTION

[0016] Against this background, it is an object of the present invention to provide a new and improved vehicle safety and / or driver assistance method, an arrangement and a data processing program product using an image capture system for a road vehicle, in particular using image capture systems that can be connected via a wireless data connection, in particular using one - in particular camera-based - rearview system or several - in particular camera-based - rearview systems, in particular for virtually expanding the field of view of such an image capture system for object detection, in particular in the case of restrictions of the field of view of the image capture system, such as restrictions due to technical limitations, distances or visual obstructions.

[0017] According to the present invention, the following is provided: - a vehicle safety and / or driver assistance method using a first image acquisition system with a first field of view according to claim 1; - a vehicle safety and / or driver assistance method using image data from data-connected - in particular camera-based - image acquisition systems mounted on at least one vehicle, according to claim 2; - a vehicle safety and / or driver assistance method in which a first - in particular camera-based - rearview image capture system with a first field of view according to claim 9 is used; - a vehicle safety and / or driver assistance device which is particularly suitable for carrying out an aforementioned method according to claim 10 and / or claim 11; - a data processing program product suitable for carrying out a vehicle safety and / or driver assistance method and suitable for carrying out an aforementioned method, in particular in an aforementioned arrangement, in particular a data processing program product suitable for carrying out a vehicle safety and / or driver assistance method using a first image acquisition system as specified in claim 15; - a data processing program product suitable for implementing a vehicle safety and / or driver assistance method using a first - in particular camera-based - rearview image capture system according to claim 16; and - a data structure for use in a vehicle safety and / or driver assistance arrangement and / or for use in a vehicle safety and / or driver assistance method according to claim 17. Advantageous or preferred embodiments of the invention are specified in the dependent claims

[0018] According to a first aspect, the present invention provides a vehicle safety and / or driver assistance method using a first image acquisition system with a first field of view, comprising the following steps: acquiring speed data; generating first field of view depth data and / or object detection distance data; evaluating, based on the acquired speed data, whether a minimum field of view depth criterion and / or a safety criterion of the minimum object distance is met; in the event that the minimum field of view depth criterion and / or the safety criterion of the minimum object distance is not met, combining image data from the first image acquisition system with image data from the second image acquisition system; evaluating, based on the combined image data, whether a minimum field of view depth criterion and / or a safety criterion of the minimum object distance is met;and depending on the evaluation of whether the criteria are met or not met, outputting and / or displaying safety data, the safety data indicating whether it is safe or unsafe to perform a particular vehicle maneuver;

[0019] The above-mentioned method according to the first aspect may be configured to operate with or in other aspects or embodiments of the invention as described in this document, including use in an arrangement according to any of the aspects or embodiments as described in this document, and including use of the method in the form of a data processing program product according to any of the aspects or embodiments as described in this document.

[0020] According to a further aspect, the present invention provides a vehicle safety and / or driver assistance method using image data from data-connected - in particular camera-based - image acquisition systems, in particular using image data from data-connected rearview image acquisition systems mounted on at least one vehicle, in particular a method according to the first aspect of the invention, comprising the steps of: acquiring image data with a first - in particular camera-based - image acquisition system, in particular with a - in particular camera-based - rearview image acquisition system; applying an image evaluation algorithm based on artificial intelligence and / or neural networks to the image data to generate first field of view depth data and / or object detection distance data; evaluating whether a minimum field of view depth criterion and / or a minimum object distance criterion is met;In the event that the minimum field of view depth criterion and / or the minimum object distance criterion is not met, receiving second image data via a data connection from at least one second - in particular camera-based - image acquisition system, in particular from at least one second - in particular camera-based - rearview image acquisition system that has a second field of view; combining the image data from the first image acquisition system and the image data from the second image acquisition system; and automatically evaluating, on the basis of the combined image data, whether the performance of a specific vehicle maneuver is safe or unsafe, in particular using a software algorithm for evaluating safety criteria, in particular using an algorithm based on artificial intelligence and / or a neural network for evaluating safety criteria.

[0021] The above-mentioned method according to the second aspect may be configured to operate with or in other aspects or embodiments of the invention as described in this document, including use in an arrangement according to any of the aspects or embodiments as described in this document, and including use of the method in the form of a data processing program product according to any of the aspects or embodiments as described in this document.

[0022] According to one aspect of the invention, in a method according to the first or second aspect as described above, as well as for use in any other embodiment of the invention as described in this document, including in an embodiment in the form of a data processing program product, an artificial intelligence-based and / or neural network-based image evaluation algorithm AI comprises a machine learning model in the form of a neural network, in particular in the form of a transversal neural network and / or in the form of a convolutional neural network, correlated with a generative pre-trained transformer.

[0023] According to one aspect of the invention, in a method according to one aspect as previously described, as well as for use in any other embodiment of the invention as described in this document, including in an embodiment in the form of a data processing program product, such a method comprises the following steps: acquiring speed limit data; correlating the acquired speed limit data with acquired first vehicle speed data; deriving safety criteria data based on the correlated speed data; and correlating the safety criteria data with the field of view depth data and / or the object distance data.

[0024] According to one aspect of the invention, in a method according to one aspect as described above, as well as for use in any other embodiment of the invention as described in this document, including in an embodiment in the form of a data processing program product, such a method comprises the steps of: acquiring first image data from a first image capture device or from a plurality of first image capture devices included in the first image capture system; and if the minimum field of view depth criterion and / or the minimum object distance safety criterion is not met, acquiring second image data from a second image capture device or from a plurality of second image capture devices included in the second image capture system.

[0025] According to one aspect of the invention, in a method according to one aspect as described above, as well as for use in any other embodiment of the invention as described in this document, including in one embodiment in the form of a data processing program product, the safety criteria data bd derived on the basis of the correlated speed data is minimum safe braking distance data, in particular data representing a minimum safe braking distance between a first vehicle on a first lane and another vehicle on the same lane or on a different lane, in particular a method and / or data suitable for avoiding a collision while a first vehicle is performing a vehicle maneuver to change from a first lane to a different lane.

[0026] According to one aspect of the invention, in a method according to one aspect as described above, as well as for use in any other embodiment of the invention as described in this document, including in an embodiment in the form of a data processing program product, such a method comprises the steps of receiving in the first vehicle, in a first wireless communication unit included in the first vehicle, image data of a second image acquisition system transmitted by a second wireless communication unit included in a second vehicle.

[0027] According to one aspect of the invention, in a method according to the first or second aspect as described above, as well as for use in any other embodiment of the invention as described in this document, including in an embodiment in the form of a data processing program product, speed limit data are acquired on the basis of further image data acquired by a further image acquisition device and / or on the basis of digital map data, in particular on the basis of further image data acquired by a further image acquisition device included in the first vehicle and / or on the basis of digital map data stored in a data storage unit of the first vehicle.

[0028] According to a further aspect, the present invention provides a vehicle safety and / or driver assistance method using a first - in particular camera-based - rearview image acquisition system with a first field of view, in particular a method according to one of the previously described aspects or with one of the features of one of the previously described aspects and for use in any other embodiment of the invention as described in this document, including in an embodiment in the form of a data processing program product, the method comprising the following steps: Acquiring image data using the first - in particular camera-based - rearview image acquisition system; Acquiring speed data of the first vehicle;Applying a field of view depth detection algorithm and / or an object detection distance algorithm to the image data to generate first field of view depth data and / or object detection distance data; correlating the speed data of the first vehicle with the field of view depth data and / or the object distance data; evaluating from the correlation whether a minimum field of view depth criterion and / or a minimum object distance safety criterion is met; in the event that the minimum field of view depth criterion and / or the minimum object distance safety criterion is not met, receiving second image data using a second—in particular camera-based—rearview image capture system with a second field of view; combining image data from the first—in particular camera-based—rearview image capture system and image data from the second—in particular camera-based—rearview image capture system;and evaluating, based on the combined image data, whether a minimum field of view depth criterion and / or a minimum object distance safety criterion is met; and depending on the evaluation of whether the criteria are met or not met, outputting and / or displaying safety data, wherein the safety data indicates whether it is safe or unsafe to perform a particular vehicle maneuver.

[0029] According to a further aspect, the present invention provides a vehicle safety and / or driver assistance system, wherein the at least one image acquisition system is a camera monitoring system (CMS), in particular a rearview camera monitoring system. Consequently, the image data is image data of the camera monitoring system.

[0030] According to a further aspect, the present invention provides a vehicle safety and / or driver assistance arrangement which is particularly suitable for carrying out a method according to one of the previously described aspects, wherein the arrangement comprises: a first image acquisition system, in particular a rearview image acquisition system, with a first field of view; one or more speed data acquisition units; an object distance detection module; a safety criteria data evaluation module; a wireless communication unit suitable for receiving image data; an image data combination module suitable for combining image data from a first image acquisition system and at least one second or further image acquisition system; and a safety data output interface which is in data communication with the safety criteria data evaluation module.

[0031] Such an arrangement may be configured to carry out a method according to any of the previously described embodiments as well as to incorporate any technical features of any other embodiment of the invention as described in this document and / or to carry out an embodiment in the form of a data processing program product as described in this document.

[0032] According to a further aspect, the present invention provides a vehicle safety and / or driver assistance arrangement that is particularly suitable for implementing a method according to one of the previously described aspects, wherein the arrangement comprises: a first—in particular camera-based—rearview image acquisition system with a first field of view; a first vehicle speed data acquisition unit; a first speed limit data acquisition unit; an object distance detection module with an artificial intelligence module; a safety criteria data evaluation module; a wireless communication unit configured to receive image data; an image data combination module configured to combine image data from a first—in particular camera-based—rearview image acquisition system and at least one second—in particular camera-based—rearview image acquisition system or another image acquisition system;and a safety data output interface that communicates with the safety criteria data evaluation module.;

[0033] According to one aspect of the invention, in any of the previously described aspects or arrangements, as well as in the other embodiments of the invention described in this document, the first image capture system comprises a first image capture device or a plurality of first image capture devices.

[0034] The aforementioned arrangement can be configured to carry out a method according to any embodiment as described above, as well as to incorporate any technical features of any other embodiment of the invention as described in this document and / or to carry out an embodiment in the form of a data processing program product as described in this document.

[0035] According to one aspect of the invention, in any of the aspects or arrangements described above, as well as in the other embodiments of the invention described in this document, the arrangement comprises a speed limit data acquisition unit arranged to acquire speed limit data based on image data acquired by an image acquisition system and / or based on digital map data.

[0036] According to a further aspect, the present invention provides a data processing program product suitable for implementing a vehicle safety and / or driver assistance method according to one of the previously described aspects and comprising instructions which, when the program is executed by a data processing unit, cause the data processing unit to execute program steps suitable for implementing a method according to one of the previously described aspects, in particular in an arrangement according to one of the previously described aspects.The aforementioned data processing program product may be configured to perform a method according to any embodiment as described in this document, as well as to incorporate any technical features of any other embodiment of the invention as described in this document, and may be executed in any arrangement as described in this document.

[0037] According to a further aspect, the present invention provides a data processing program product suitable for implementing a vehicle safety and / or driver assistance method according to one of the previously described aspects and comprising instructions which, when the program is executed by a data processing unit, cause the data processing unit to execute the following program steps, in particular according to a method according to one of the previously described aspects: a first program step of reading first image data from the first image acquisition system; a second program step of reading object distance data from an object distance acquisition module; a third program step of reading speed data from a speed data acquisition unit; a fourth program step of evaluating whether a minimum safety criterion is met;a fifth program step of reading, if the minimum safety criterion is not met, second image data from a first communication unit, in particular second image data transmitted by a second communication unit that is in data communication with a second image acquisition system; a sixth program step of combining the image data from the first image acquisition system with the image data from the second image acquisition system; and a seventh program step of evaluating, based on the combined image data, whether a minimum safety criterion is met; and an eighth program step of outputting and / or displaying safety data, depending on the evaluation of whether the criterion is met or not met.

[0038] The above-mentioned data processing program product can be configured to perform a method according to any of the embodiments described in this document, as well as to include any technical features of any other embodiment of the invention described in this document, and to be executed in any of the arrangements described in this document.

[0039] According to a further aspect, the present invention provides a data processing program product suitable for implementing a vehicle safety and / or driver assistance method according to one of the previously described aspects, comprising instructions which, when the program is executed by a data processing unit, cause the data processing unit to carry out the following program steps, in particular according to a method according to one of the previously described aspects: a first program step of reading out first image data from the first - in particular camera-based - rearview image acquisition system;a second program step of reading object distance data from an object distance detection module and / or applying a field of view depth detection algorithm and / or an object detection distance algorithm containing an artificial intelligence algorithm implemented in a software-based artificial intelligence module to the image data to generate first field of view depth data and / or object detection distance data; a third program step of reading first vehicle speed data and / or speed limit data from one speed data detection unit or from a plurality of speed data detection units; a fourth program step of correlating the first vehicle speed data and / or speed limit data with the field of view depth data and / or the object distance data;a fifth program step of evaluating from the correlation whether a minimum field of view depth criterion and / or a safety criterion of the minimum object distance is met; a sixth program step of reading, if the minimum safety criteria are not met, second image data from a first communication unit, in particular second image data transmitted by a second communication unit that is in data communication with a second - in particular camera-based - rearview image capture system; a seventh program step of combining image data from the first - in particular camera-based - rearview image capture system and image data from the second - in particular camera-based - rearview image capture system; an eighth program step of evaluating, based on the combined image data, whether a minimum field of view depth criterion and / or a safety criterion of the minimum object distance is met;and a ninth program step of outputting and / or displaying safety data, depending on the evaluation of whether the criteria are met or not met, wherein the safety data indicates whether the performance of a particular vehicle maneuver (VM) is safe or unsafe.;

[0040] The above-mentioned data processing program product can be configured to perform a method according to any of the embodiments described in this document, as well as to include any technical features of any other embodiment of the invention described in this document, and to be executed in any of the arrangements described in this document.

[0041] According to a further aspect, the present invention provides a data structure for use in a vehicle safety and / or driver assistance arrangement according to one of the previously described aspects and / or for use in a vehicle safety and / or driver assistance method according to one of the previously described aspects, in particular a data structure generated in a vehicle safety and / or driver assistance arrangement according to one of the previously described aspects, and / or a data structure generated by a method according to one of the previously described aspects or by a method comprising one or more of the features of one of the previously described aspects, wherein such a data structure comprises combined image data comprising first image data generated by a first on-vehicle image acquisition system having a first field of view, and second image data,which are generated by a second on-board image acquisition system with a second field of view, wherein such a data structure and / or combined image data are adapted as input data for an on-board field of view depth detection and / or object distance detection module; and / or such a data structure and / or combined image data are suitable as input data for an on-board safety criteria evaluation unit.

[0042] The embodiments described above can be combined with one another as desired, if appropriate. Further possible embodiments, further refinements, and implementations of the invention also include combinations of features of the invention described herein not explicitly mentioned in relation to the embodiments. In particular, those skilled in the art will also add individual aspects as improvements or additions to the respective basic form of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] For a more complete understanding of the invention and its advantages, exemplary embodiments of the invention are explained in more detail in the following description with reference to the accompanying figures, in which like reference numerals designate like parts and in which the following is shown: Fig. 1 a schematic representation of a first traffic situation with several vehicles whose image capture systems have a sufficient field of view; Fig. 2 a schematic representation of a second traffic situation with several vehicles whose image capture systems have an insufficient field of view; Fig. 3 a schematic representation of a third traffic situation with several vehicles whose image capture systems have an insufficient field of view due to visual obstructions; Fig. 4 a schematic representation of a fourth traffic situation with several vehicles whose image capture systems have an insufficient field of view due to visibility obstructions caused by the road shape; Fig. 5 is a schematic representation of a fifth traffic situation with several vehicles and with an alternative arrangement of image acquisition systems having an insufficient field of view; Fig. 6 is a schematic representation of a sixth traffic situation with multiple vehicles and with an alternative arrangement of image acquisition systems providing insufficient field of view depth; Fig. 7 is a flowchart of a first part of a specific embodiment of a vehicle safety and / or driver assistance method according to the invention; Fig. 8 shows a flowchart of a second part of a specific embodiment of a vehicle safety and / or driver assistance method according to the invention; Fig. 9 a schematic representation of a specific embodiment of a vehicle safety and / or driver assistance arrangement according to the invention; Fig. 10 is a schematic representation of the creation of a specific embodiment of a data structure that combines image data from at least two image acquisition systems according to the invention; Fig. 11 is a schematic flow diagram of a first specific embodiment of a data processing program product for implementing a vehicle safety and / or driver assistance method according to the invention; and Fig. 12 is a schematic flow diagram of a second specific embodiment of a data processing program product for implementing a vehicle safety and / or driver assistance method according to the invention.

[0044] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate particular embodiments of the invention and, together with the description, serve to explain the principles of the invention. Other embodiments of the invention and many of the attendant advantages of the invention will be readily appreciated as they become better understood by reference to the following detailed description.

[0045] It is understood that common and / or well-understood elements that may be useful or necessary in a commercially viable embodiment are not necessarily depicted in order to provide a more abstract view of the embodiments. The elements of the drawings are not necessarily drawn to scale relative to one another. It is understood that certain acts and / or steps in an embodiment of a method may be described or depicted in a particular order, while those skilled in the art will understand that such specificity regarding the order is not actually required. It is also understood that the terms and expressions used in the present description have the usual meanings attributed to these terms and expressions with respect to the corresponding fields of technology, unless specific meanings are otherwise set forth herein. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] In the future, assisted and / or automated driving will be increasingly linked to vehicle-to-vehicle communication, and highway assistants, in particular, will implement assisted and / or automated vehicle maneuvers such as overtaking, starting at level 4 at the latest. The following embodiments of the invention explain in more detail how vehicle maneuvers, such as overtaking, can be autonomously supported and / or performed with maximum safety. This topic can be implemented, in particular, by the Co-programmed Partnership for Connected Automated Mobility (CCAM).

[0047] One problem with the aforementioned assisted and / or automated driving scenarios is that, when overtaking, the decision whether to overtake depends largely on the depth of field of view of the vehicle's own image capture systems, such as camera-based rearview image capture systems and / or, in particular, camera-based frontal image capture systems. Visibility plays a major role here. An overtaking maneuver may only be initiated if objects can be reliably detected and are within sufficient range. This applies to both autonomously driving decision-makers and assisted decision-makers. Even the rear or front vision device alone can support decision-making if the driver receives indications that they do not have sufficient visibility to the rear and / or forward.

[0048] To achieve this objective, images captured by on-board image capture systems—in particular, as in one possible embodiment of the invention, with camera-based image capture systems—are evaluated using software to determine how far a respective image capture system can see backward and / or forward. This can be achieved using technologies as described above with reference to the background of the invention. In particular, this objective can be achieved by implementing an AI algorithm such that the distance can be determined based on captured images or image sequences from one or more respective image capture systems, in particular on a pixel basis.In a second step, it can be determined whether – based on the visibility determination – objects can be detected that are located at a distance at which a certain intended vehicle maneuver, such as an overtaking maneuver, is permissible, i.e., the corresponding safety criteria for such a vehicle maneuver are met. In the event that this is not the case, the invention provides that the vehicle's own automated and / or assisted driving system automatically searches for vehicles in the vicinity and within range of a vehicle's own wireless communication unit, e.g.for other connectable vehicles behind or in front of the own vehicle, and establishes a wireless data connection to corresponding image capture systems on board these other vehicles in order to obtain their rear or front view information, which can be combined with the image data acquired by the image capture system(s) on board the own vehicle. Since such a data connection to other on-board image capture systems increases the field of view to the rear or front based on the own location, a field of view extension is provided based on the received image data.

[0049] With reference to Fig. Figure 1 of the drawings shows a schematic view of a first traffic situation in which several vehicles 1, 2, 3, 4 are traveling on several adjacent lanes L1, L2. One or more of the illustrated vehicles 1, 2, 3, 4 are equipped, in particular, with camera-based image capture systems CMS1, CMS2, which are mounted on at least one vehicle 1, 2, 3, 4. In Fig. 1, in particular, two vehicles 1, 2 are equipped with at least one camera-based image capture system CMS1, CMS2 per vehicle. Such camera-based image capture systems CMS1, CMS2 can be designed, in particular, as camera-based rearview image capture systems CMS1, CMS2 or can be part thereof. In particular, the camera-based image capture systems CMS1, CMS2 can be part of a so-called camera surveillance system (CMS).

[0050] Each camera-based image capture system CMS1, CMS2 comprises at least one image capture device CD1, CD2, in particular a camera device, or comprises several image capture devices CD1, CD1x, CD2, CD2x, in particular several camera devices, as described with reference to Fig. 9 is explained in more detail.

[0051] The above-mentioned vehicle(s) 1, 2 carrying such camera-based image capture systems CMS1, CMS2 also each include a wireless communication unit A1, A2 configured for wireless data transmission to corresponding other communication units A1, A2 of other vehicles 1, 2. Not all vehicles 1, 2, 3, 4 participating in a traffic situation need to be equipped with such camera-based image capture systems CMS1, CMS2 and / or wireless communication units A1, A2.

[0052] As in Fig. 1, a first camera-based image capture system CMS1 of a first vehicle 1 - in particular a first camera-based rearview image capture system CMS1 - has a first field of view (FoV) F1, and the depth of this first field of view F1 corresponds to a maximum object detection distance dOD at which objects can still be detected within this first field of view F1 by the first camera-based image capture system CMS1 of a first vehicle 1.

[0053] At least the first vehicle 1 further comprises a further camera-based image acquisition system CMSx, in particular a further camera device, which is designed to acquire speed limit data, in particular to acquire and evaluate traffic signs or other speed limit displays, as in Fig. 1 is shown schematically.

[0054] In the situation according to Fig. 1, it is assumed that the above-mentioned maximum object detection distance dOD is greater than a minimum object detection distance (dODm) corresponding to a minimum first field of view F1m, both of which are derived from or correspond to minimum safety criteria, as further explained in the following description and with reference to the following drawings.

[0055] Since in the situation according to Fig. 1 a minimum safety criterion dODm is met, a certain vehicle maneuver VM is deemed safe for the first vehicle 1, in Fig. 1 in particular an overtaking maneuver to overtake a preceding vehicle 4 in the same lane L1 by changing to the adjacent lane L2, since the distance dOD to a detected vehicle 3 behind the vehicle 1 traveling in lane L2 is greater than the minimum safety criterion dODm.

[0056] This means that in case of significant visibility of relevant objects in the surroundings of a vehicle 1 equipped with at least one camera-based image acquisition system CMS1 to detect, analyze and confirm that certain safety criteria dODm are met, no additional data input is required for such a decision and confirmation of the fulfillment of the safety criteria dODm.

[0057] In Fig. 2 shows a schematic representation of a second traffic situation in which - similar to Fig. 1 - several vehicles 1, 2, 3, 4 are driving on several adjacent lanes L1, L2. As in Fig. 1, several of the vehicles 1, 2, 3, 4 shown are also equipped with camera-based image acquisition systems CMS1, CMS2, which are attached to at least one vehicle 1, 2, 3, 4.

[0058] As in Fig. 2, a first camera-based image acquisition system CMS1 of a first vehicle 1 has a first field of view F1. In the situation of Fig. 2, however, the depth of this first field of view F1, which corresponds to a maximum object detection distance dOD for which objects within this first field of view F1 can still be detected by the first camera-based image capture system CMS1 of a first vehicle 1, is now smaller than a minimum object detection distance dODm, which corresponds to a minimum first field of view F1m, both of which in turn are derived from or correspond to minimum safety criteria. In Fig. 2, this may be due to technical and / or optical limitations of the first camera-based image acquisition system CMS1 and / or due to excessive distances between the respective vehicles 1, 2, 3, 4, and / or due to the respective speeds and speed limits of the respective vehicles 1, 2, 3, 4, as will be explained in more detail later.

[0059] Alternatively or additionally, an insufficient depth of field of view may be caused by obstructions to the view VO in the area of ​​vehicle 1, as in Fig. 2, and / or by the distances between the respective vehicles 1, 2, 3, 4 or by the respective speeds and speed limits of the respective vehicles 1, 2, 3, 4. As shown in Fig. 3, a first camera-based image acquisition system CMS1 of a first vehicle 1 has a first field of view F1. However, in the situation of Fig. 3 however, due to visibility obstructions that may be caused by environmental conditions such as weather conditions (e.g. rain, snow, fog), lighting conditions (e.g. darkness, shadows, glare) or other visibility obstructions (e.g. smoke, dust), the depth of this first field of view F1, which corresponds to a maximum object detection distance dOD at which objects within this first field of view F1 can still be detected by the first camera-based image capture system CMS1 of a first vehicle 1, is in turn smaller than a minimum object detection distance dODm, which corresponds to a minimum first field of view F1m, both of which in turn are derived from or correspond to minimum safety criteria.

[0060] This means that the first camera-based image acquisition system CMS1 of the first vehicle 1 in given traffic situations and at the given vehicle speeds and speed limits according to Fig. 2 or Fig. 3 provides insufficient field of view depth to detect and analyze relevant objects and to decide whether safety criteria are met to safely perform an intended vehicle maneuver VM.

[0061] In such a situation according to Fig. 2 or Fig. 3, the first wireless communication unit A1 of the first vehicle 1 establishes a wireless data connection and data exchange with a second wireless communication unit A2 of a second vehicle 2 within the transmission range of the wireless communication unit A1 in order to receive image data acquired by a second camera-based image acquisition system CMS2 mounted on the second vehicle 2 and having a second field of view F2, in particular by a second camera-based rearview image acquisition system CMS2.

[0062] The received image data, which are captured by a second camera-based image capture system CMS2 mounted on the second vehicle 2, serve to virtually expand the field of view of the first second camera-based image capture system CMS2 in order to ensure object detection and / or object distance detection in a sufficient distance range from the first vehicle 1 in order to be able to decide whether safety criteria are met in order to be able to safely carry out an intended vehicle maneuver VM, in particular a maneuver to overtake a preceding vehicle 4 in the same lane L1 by moving into the adjacent lane L2.

[0063] In Fig. 4 shows a fourth traffic situation in which - similar to Fig. 1 and Fig. 2 - several vehicles 1, 2, 3, 4 are driving on several adjacent lanes L1, L2. As in Fig. 1, Fig. 2 and Fig. 3, several of the vehicles 1, 2, 3, 4 shown are also equipped with camera-based image acquisition systems CMS1, CMS2, which are attached to at least one vehicle 1, 2, 3, 4.

[0064] Again similar to the situations of Fig. 2 and Fig. 3, the depth of the first field of view F1, which corresponds to a maximum object detection distance dOD for which objects within this first field of view F1 can still be detected by the first camera-based image capture system CMS1 of a first vehicle 1, is smaller than a minimum object detection distance dODm, which corresponds to a minimum first field of view F1m, both of which in turn are derived from or correspond to minimum safety criteria. In the situation according to Fig. 4, this is caused by visual obstructions due to curves in the road in the area of ​​the vehicle 1, which cause this visual obstruction for the first camera-based image capture system CMS1 of the first vehicle 1.

[0065] This means that the first camera-based image acquisition system CMS1 of the first vehicle 1 for the given road shape according to Fig. 4 has an insufficient field of view depth to detect and analyze relevant objects such as the vehicle 3 and to decide whether safety criteria are met to safely perform an intended vehicle maneuver VM, in particular an overtaking maneuver to overtake a preceding vehicle 4 on the same lane L1 by moving into the adjacent lane L2.

[0066] To overcome this obstacle in the situation according to Fig. 4, the first wireless communication unit A1 of the first vehicle 1 in turn establishes a wireless data connection and a data exchange with a second wireless communication unit A2 of a second vehicle 2 within the transmission range of the wireless communication unit A1 in order to receive image data acquired by a second camera-based image acquisition system CMS2 which is attached to the second vehicle 2 and has a second field of view F2, in particular by a second camera-based rearview image acquisition system CMS2.

[0067] Even in the situation according to Fig. 4, the received image data captured by a second camera-based image capture system CMS2 mounted on the second vehicle 2 are used to virtually extend the field of view of the first second camera-based image capture system CMS2 to ensure object detection and / or object distance detection in a sufficient distance range from the first vehicle 1 in order to be able to decide whether safety criteria are met to safely perform an intended vehicle maneuver VM.

[0068] In Fig. Figure 5 shows a fifth traffic situation with several vehicles 1, 2, 3 and with an alternative arrangement of image capture systems CMS1, CMS2, which have an insufficient field of view. A first camera-based image capture system CMS1 of a first vehicle 1, in turn, has a first field of view F1. In the arrangement of Fig. 5, however, this first camera-based image capture system CMS1 of a first vehicle 1 is now directed in a forward direction, in particular as a first camera-based front-view image capture system CMS1, ie in the direction of travel of the vehicle 1, in contrast to each of the situations of Fig. 1 to Fig. 4, in which a first camera-based image capture system CMS1 of a first vehicle 1 is directed in a reverse direction, ie opposite to the direction of travel of the vehicle 1. In the situation of Fig. 5, the depth of the first field of view F1, which corresponds to a maximum object detection distance dOD for which objects within this first field of view F1 can still be detected by the first camera-based image acquisition system CMS1 of a first vehicle 1, is in turn smaller than a minimum object detection distance dODm, which corresponds to a minimum first field of view F1m, both of which are in turn derived from or correspond to minimum safety criteria. In Fig. 5 this may be due to technical and / or optical limitations of the first camera-based image acquisition system CMS1 and / or due to excessive distances between the respective vehicles 1, 3, and / or due to the respective speeds and speed limits of the respective vehicles 1, 2, 3, as will be explained in more detail later.

[0069] To overcome this obstacle in the situation according to Fig. 5, the first wireless communication unit A1 of the first vehicle 1 in turn establishes a wireless data connection and a data exchange with a second wireless communication unit A2 of a second vehicle 2 in the transmission range of the wireless communication unit A1 - here to a second vehicle 2 traveling in front of the first vehicle 1 - in order to receive image data which is captured by a second camera-based image capture system CMS2 which is attached to the second vehicle 2 and has a second field of view F2, in particular by a second camera-based front view image capture system CMS2.

[0070] Fig. Figure 6 shows a schematic representation of a sixth traffic situation with several vehicles and with an alternative arrangement of image acquisition systems with insufficient field of view depth, wherein the same reference numerals as in Fig. 1 to Fig. 5, which have the same or a similar meaning. Again, a traffic situation is shown with several vehicles 1, 2, 3 and with an alternative arrangement of image capture systems CMS1, CMS2, which have an insufficient field of view depth. A first camera-based image capture system CMS1 of a first vehicle 1 traveling in a first lane L1, in turn, has a first field of view F1. In the arrangement of Fig. 6, this first camera-based image capture system CMS1 of a first vehicle 1 is again directed in a forward direction, in particular it is a first camera-based front-view image capture system CMS1, ie directed in the direction of travel of the vehicle 1. However, such a first camera-based image capture system CMS1 of a first vehicle 1 can additionally or alternatively be directed in a reverse direction, ie opposite to the direction of travel of the vehicle 1, as previously described. In the situation of Fig. 6, the direction and / or depth of the first field of view F1 is again not sufficient (smaller or directed in a different direction) to capture all relevant objects.

[0071] To overcome this obstacle in the situation according to Fig. 6, the first wireless communication unit A1 of the first vehicle 1 establishes a wireless data connection and a data exchange with a second wireless communication unit A2 of a second vehicle 2 traveling in an adjacent lane L2, within the transmission range of the wireless communication unit A1 - here to a second vehicle 2 traveling in front of the first vehicle 1 - in order to receive image data captured by a second camera-based image capture system CMS2, which is attached to the second vehicle 2 and has a second field of view F2, in particular by a second camera-based front-view image capture system CMS2 and / or by a second camera-based rear-view image capture system CMS2. In the specific embodiment of Fig. 6, such a second camera-based front and / or rear view image acquisition system CMS2 is directed towards an adjacent lane L3 on the opposite side of the vehicle, i.e. a lane L3 opposite the lane L1 with respect to the (central) lane L3, in order to detect objects, i.e. vehicles 3, that may cause a safety risk or even a collision for the vehicle 2 and other adjacent vehicles, such as the aforementioned vehicle 1.

[0072] Such an arrangement and such a procedure and / or corresponding data are again - similar to Fig. 1 to Fig. 5 - suitable for avoiding a collision while a first vehicle 1 performs a vehicle maneuver VM to change from a first lane (L1) to another lane L2, L3, such as during the overtaking maneuver of a vehicle, whereby it moves to another lane L2 to initiate the overtaking maneuver, as shown previously, or moves back to the original first lane L1 to terminate the overtaking maneuver, as shown in Fig. 6 shown.

[0073] Fig. 7 shows a flowchart of a first part of a particular embodiment of a vehicle safety and / or driver assistance method according to the invention, and Fig. 8 shows a flowchart of a second part of a particular embodiment of a vehicle safety and / or driver assistance method according to the invention. Fig. The first part of an embodiment of a vehicle safety and / or driver assistance method according to the invention shown in Figure 7 comprises the following steps using the example of a vehicle maneuver in the form of an overtaking maneuver: - Capture the first image data of the first CMS (CMS1) - Detection distance for object detection (dOD) - Determination of speed data, consisting of the following steps: ◯ Determination of own vehicle speed data ◯ Detecting speed limit data - Calculation of braking distance (bd) = minimum object distance (dODm) - Evaluate whether distance for object detection dOD > braking distance bd If the result of this first evaluation is positive = yes: - Evaluate whether an object (3) is detected within the minimum object distance dODm If the result of this second evaluation is positive = yes: ▪ Make a decision that overtaking maneuver (VM) is not allowed If the result of this second assessment is negative = no: ▪ Make the decision that the overtaking maneuver (VM) is allowed. If the result of the first evaluation is negative = no: - Detection of a second connectable image acquisition system CMS2 of a second, different vehicle 2 within the object detection distance dOD

[0074] The second part of a concrete embodiment of a vehicle safety and / or driver assistance method according to the invention according to Fig. 8, using the example of a vehicle maneuver in the form of an overtaking maneuver, includes the following further steps: - Check whether the range of the wireless data transmission of the vehicle's wireless communication unit A1 < object detection distance dOD If the result of this third evaluation is negative = no: ◯ Decision that the overtaking maneuver (VM) is not allowed If the result of this third evaluation is positive = yes: ◯ Check whether a connectable image capture system CMS2 of a second, different vehicle 2 on the same lane L1 can be detected If the result of this fourth evaluation is negative = no: ▪ Make a decision that the overtaking maneuver (VM) is not allowed If the result of this fourth assessment is positive = yes: ▪ Receiving second image data from the second image acquisition system CMS2 of the second, other vehicle 2 ▪ Detecting the object detection distance (dOD) using the received second image data from the second image acquisition system CMS2 of the second, other vehicle 2 ▪ Evaluate whether object detection distance dOD > braking distance bd If the result of this fifth evaluation is positive = yes: - Evaluate whether an object (3) is detected within the minimum object distance dODm If the result of this sixth evaluation is positive = yes: - Make a decision that the overtaking maneuver (VM) is not allowed If the result of the second sixth evaluation is negative = no: - Make the decision that the overtaking maneuver (VM) is allowed If the result of the fifth evaluation is negative = no:

[0075] Detection of another connectable image acquisition system CMSx of another vehicle within the object detection distance dOD

[0076] A detailed calculation example of the above-mentioned embodiment of the invention is described below with reference to the above-mentioned figures for an intended vehicle maneuver to overtake a preceding vehicle 4: Speed ​​of the first vehicle 1 (car in the right lane L1): 80 km / h Speed ​​of vehicle 3 (vehicle in the left lane L2): 200 km / h Difference in speed between vehicle 1 and vehicle 3: 120 km / h that means: 30 seconds / km which results in a distance of 200 m in 6 seconds which results in a required visibility of 200 m day and night to allow overtaking.

[0077] The road view to the rear may be limited to 170m due to curves / night / rain / etc., as in Fig. 2, Fig. 3 or Fig. 4. Therefore, it is necessary to search for another, second vehicle 2 with a networked camera-based image acquisition system CMS2 within a radius of at least 50 m and to establish a wireless data connection to such a second vehicle 2.

[0078] By combining the image data from the first vehicle 1 and the second vehicle 2 via a wireless data connection, a virtually extended field of view of at least 170 m + 50 m = 220 m is achieved.

[0079] This provides a sufficient field of vision for the rear view of vehicle 1 and an overtaking maneuver, ie a vehicle maneuver VM, can be released if no object is visible in the overtaking lane L2 within a safety area resulting from the current braking distance(s) of the respective relevant vehicle(s) 1, 3.

[0080] From the table below, an expert can determine the braking distance. At an absolute or relative (difference) speed of 120 km / h, the respective braking distance is 144 m. The driver's reaction time is not included in this calculation.

[0081] The braking distance can be calculated using the following formula: Normal braking distance ≈(driven speed ÷ 10)×(driven speed ÷ 10).

[0082] This results in the following braking distances: Geschwindigkeit Einzelbremsweg 30 km / h 9 m 50 km / h 25 m 70 km / h 49 m 100 km / h 100 m 130 km / h 169 m

[0083] The visibility determination is based on a determination of speed data, which may include traffic sign recognition, as described in the Fig. 2 to Fig. 6. By comparing the own vehicle speed of the first vehicle 1 and the maximum permissible speed derived from the traffic sign recognition, a dynamic calculation is possible. Alternatively or additionally, speed data can be derived from digital map data, as described in the following with reference to Fig. 9 is shown and explained.

[0084] Fig. 9 shows schematically a specific embodiment of a vehicle safety and / or driver assistance arrangement according to the invention. In particular Fig. 7 schematically shows functional elements, units and modules of a vehicle safety and / or driver assistance arrangement, which are particularly suitable for carrying out a method according to Fig. 7 and Fig. 8 and / or according to the following description and Fig. 10 and / or Fig. 11 and / or Fig. 12 are suitable. The aforementioned functional elements, units, and modules can be implemented as hardware elements, units, and modules or as software elements, units, and modules or as a combination of both, hardware and software.

[0085] The specific embodiment according to Fig. 9 comprises a vehicle safety and / or driver assistance arrangement of a first vehicle 1, which comprises at least the following functional elements, units and / or modules: a first image acquisition system CMS1 with a first field of view F1, as in Fig. 1 to Fig. 6; a first object distance detection module ODD1, which can be contained in a first image acquisition system CMS1 or alternatively in a first data processing unit DPU1 of the vehicle 1; at least one second or further image acquisition system CD11; a plurality of speed data acquisition units SLDU, VSDU; a first data processing unit DPU1 of the vehicle 1; an image data combination module IDCM, which is suitable for combining image data from a plurality of image acquisition systems CMS1, CMS2, CMSx; a braking distance calculation module BDCM; a distance comparison module DCM; a safety criteria data evaluation module SCEM; a first wireless communication unit A1, which is suitable for receiving image data; and a safety data output interface SDOI, which is in data connection with the safety criteria data evaluation module SCEM.

[0086] The first image acquisition system CMS1 can comprise a first image acquisition device CD1 or a plurality of first image acquisition devices CD1, CD1x, in particular image acquisition devices CD1, CD1x configured as camera devices, in particular as camera-based rearview devices and / or as camera-based front-view devices. The plurality of speed data acquisition units SLDU, VSDU can in particular comprise a first vehicle speed data acquisition unit VSDU and a first speed limit data acquisition unit SLDU.The speed limit data acquisition unit SLDU may be adapted to acquire speed limit data on the basis of image data acquired by an image acquisition system CMSx in data communication with the speed limit data acquisition unit SLDU and / or on the basis of digital map data DM, such digital map data DM being stored in a digital map data storage unit DM in data communication with the speed limit data acquisition unit SLDU.

[0087] The first object distance detection module ODD1 may comprise an artificial intelligence AI module, in particular an artificial intelligence according to a technology as previously described in the background of the invention, which may comprise a software-implemented AI algorithm that may comprise a machine learning model in the form of a neural network, in particular in the form of a transverse neural network and / or in the form of a convolutional neural network, correlated with a generative pre-trained transformer that uses special algorithms to find patterns in data sequences, in particular in image data representing images captured by one or more on-board image capture systems and / or in image data representing image sequences captured by one or more on-board image capture systems.

[0088] The specific embodiment according to Fig. 9 also shows functional elements, units and / or modules of a further, second vehicle 2, which is in wireless data connection with the aforementioned first vehicle 1, in particular functional elements, units and / or modules of a further, second vehicle safety and / or driver assistance arrangement of a second vehicle 2, which comprises at least the following functional elements, units and / or modules: a second image acquisition system CMS2 with a second field of view F2, as in Fig. 1 to Fig. 6; a second object distance detection module ODD2, which may be included in a second image acquisition system CMS2 or alternatively in a second data processing unit DPU2 of the vehicle 2; a second data processing unit DPU2 of the vehicle 2; and a second wireless communication unit A2, which is configured to transmit image data.

[0089] The second image acquisition system CMS2 may comprise a second image acquisition device CD2 or a plurality of second image acquisition devices CD2, CD2x, in particular image acquisition devices CD2, CD2x, which are designed as camera devices, in particular as camera-based rear-view devices and / or as camera-based front-view devices. The second object distance detection module ODD2 may also comprise an artificial intelligence AI module (in Fig. 9 not shown), which may comprise a software-implemented AI algorithm, in particular an artificial intelligence according to a technology as previously described in the background of the invention, which may comprise a machine learning model in the form of a neural network, in particular in the form of a transversal neural network and / or in the form of a convolutional neural network, correlated with a generative pre-trained transformer which, with the aid of special algorithms, finds patterns in data sequences, in particular in image data representing images recorded by one or more on-board image acquisition systems, and / or in image data representing image sequences recorded by one or more on-board image acquisition systems.

[0090] This shows Fig. 9 shows, in particular schematically, a specific embodiment of a vehicle safety and / or driver assistance arrangement according to the invention, which is used to carry out a method according to Fig. 7 and Fig. 8 is suitable, comprising: a first camera-based rearview image acquisition system CMS1 with a first field of view F1; a first vehicle speed data acquisition unit VSDU; a first speed limit data acquisition unit SLDU; an object distance detection module ODD1; a safety criteria data evaluation module SCEM; a wireless communication unit A1 suitable for receiving image data; an image data combination module IDCM, which is suitable for combining image data of a first camera-based rearview image acquisition system CMS1 and at least one second camera-based rearview image acquisition system or another image acquisition system CMS2, CD11; and a safety data output interface SDOI, which is in data communication with the safety criteria data evaluation module SCEM; wherein the first image acquisition system CMS1 comprises a first image acquisition device CD1 or a plurality of first image acquisition devices CD1, CD1x; and the vehicle safety and / or driver assistance arrangement further comprises a speed limit data acquisition unit SLDU configured to acquire speed limit data based on image data acquired by an image acquisition device CMSx and / or based on digital map data DM.

[0091] Fig. 10 schematically shows the creation of a specific embodiment of a data structure that combines first image data ID1 and second image data ID2 from at least two image acquisition systems CMS1, CMS2 of at least two vehicles 1, 2 according to the invention, using a wireless data connection based on respective wireless communication units A1, A2 of the vehicles 1, 2. The data structure comprises combined image data ID1+ID2 containing image data generated by a first on-board image acquisition system CMS1 with a first field of view F1 and image data generated by a second on-board image acquisition system CMS2 with a second field of view F2, wherein such a data structure and / or combined image data ID1+ID2 are adapted as input data for an on-board depth detection of the field of view F1, F2 and / or an object distance detection module ODD1, ODD2.and / or such a data structure and / or combined image data ID1+ID2 are adapted as input data for a vehicle-specific safety criteria evaluation module SCEM, each of which can be in data communication with an image data combination module IDCM adapted to generate such a data structure and / or combined image data ID1+ID2, and such an image data combination module IDCM is implemented in a first data processing unit DPU1 of the first vehicle 1 as a software-implemented module or as a hardware-implemented module.

[0092] Such a data structure can, in particular, comprise an image content data block composed of combined image data ID1+ID2, wherein the size of the data block composed of combined image data ID1+ID2 is smaller than the sum of the size of a first data block with image content of the first image data ID1 and a second data block with image content of the second image data ID2. Such a data structure can, in particular, additionally comprise data derived from the first field of view data F1 of the first image acquisition system CMS1 and data derived from the second field of view data F2 of the second image acquisition system CMS2.

[0093] Fig. 11 shows a schematic flow diagram of a first specific embodiment of a data processing program product for implementing a vehicle safety and / or driver assistance method according to the invention. Fig. The data processing program product shown in Figure 11 can be adapted for implementing a vehicle safety and / or driver assistance method using a first image acquisition system CMS1 with a first field of view F1, as shown in the previously described figures. The data processing program product comprises instructions that, when executed by a data processing unit DPU1 of a vehicle 1, cause the data processing unit DPU1 to execute the following program steps: a first program step 101 of reading first image data from the first image acquisition system CMS1 of the vehicle 1; a second program step 102 of reading object distance data from an object distance detection module ODD1 of the vehicle 1; a third program step 103 of reading speed data from a speed data acquisition unit SLDU, VSDU; a fourth program step 104 in which it is assessed whether a minimum safety criterion bd and / or dODm, as previously described, is met; a fifth program step 105 of reading, if the minimum safety criteria bd and / or dODm are not met, second image data from a first communication unit A1 of the vehicle 1, in particular second image data transmitted by a second communication unit A2 of a second vehicle 2, which is in data connection with a second image acquisition system CMS2 of the second vehicle 2; a sixth program step 106 of combining the image data ID1 of the first image acquisition system CMS1 with the image data ID2 of the second image acquisition system CMS2 to generate combined image data ID1+ID2 and in particular a data structure containing such combined image data ID1+ID2; and a seventh program step 107, in which it is evaluated on the basis of the combined image data whether a minimum safety criterion bd, dODm, as previously described, is met; and an eighth program step 108 of outputting and / or displaying safety data, depending on the assessment of whether the criteria bd, dODm are met or not met.

[0094] Fig. 12 shows a schematic flow diagram of a second specific embodiment of a data processing program product for implementing a vehicle safety and / or driver assistance method according to the invention. Fig.The data processing program product illustrated in Figure 12 can be adapted for implementing a vehicle safety and / or driver assistance method using a first image acquisition system CMS1 with a first field of view F1, as illustrated in the previously described figures. The data processing program product comprises instructions which, when executed by a data processing unit DPU1 of a vehicle 1, cause the data processing unit DPU1 to execute the following program steps: a first program step 201 of reading the first camera-based rearview image acquisition system CMS1; a second program step 202 of reading object distance data from an object distance detection module ODD1 and / or applying a depth detection algorithm of the field of view (F1) and / or an object detection distance algorithm (dOD) to the image data, wherein one or each of the aforementioned algorithms comprises, in particular, an artificial intelligence algorithm as already described in this document and implemented in a software-based artificial intelligence module AI to generate depth data of the first field of view F1 and / or object detection distance data dOD, wherein the AI ​​algorithm implemented within the scope of the invention can be designed to comprise a generative pre-trained transformer that uses special algorithms to find patterns in image data and / or image data sequences, in particular in image data representing images captured by one or more on-board image acquisition systems,and / or in image data representing image sequences captured by one or more on-board image acquisition systems, i.e. such an AI algorithm comprises a machine learning model in the form of a neural network, in particular in the form of a transversal neural network and / or in the form of a convolutional neural network, correlated with a generative pre-trained transformer;, a third program step 203 of reading speed data of the first vehicle 1 and / or speed limit data from a speed data acquisition unit SLDU, VSDU or from a plurality of speed data acquisition units SLDU, VSDU; a fourth program step 204 in which the speed data of the first vehicle 1 and / or the speed limit data are correlated with the field of view depth data F1 and / or the object distance data dOD; a fifth program step 205, in which the correlation is used to evaluate whether a minimum field of view depth criterion F1 and / or a safety criterion of the minimum object distance dODm is met; a sixth program step 206 for reading, if the minimum safety criteria bd, dODm are not met, second image data from a first communication unit A1, in particular second image data transmitted by a second communication unit A2 which is in data connection with a second camera-based rearview image acquisition system CMS2; a seventh program step 207 of combining image data from the first camera-based rearview image acquisition system CMS1 and image data from the second camera-based rearview image acquisition system CMS2; an eighth program step 208 in which, based on the combined image data, it is evaluated whether a minimum depth criterion of the field of view (F1, F2) and / or a safety criterion of the minimum object distance dODm is met based on the combined image data ID1+ID2; and a ninth program step 209 of outputting and / or displaying safety data, depending on the assessment of whether the criterion dODm is met or not met, wherein the safety data indicate whether the execution of a specific vehicle maneuver VM is safe or unsafe.

[0095] Although specific embodiments of the invention are shown and described herein, it will be appreciated by those skilled in the art that numerous alternative and / or equivalent embodiments exist. It should be understood that the exemplary embodiment or exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration in any way. Rather, the foregoing summary and detailed description are intended to provide one skilled in the art with a convenient road map for implementing at least one exemplary embodiment; it should be understood that various changes may be made in the function and arrangement of the elements described in an exemplary embodiment without departing from the scope of the appended claims and their legal equivalents.In general, this application is intended to cover any adaptations or variations of the specific embodiments described herein.

[0096] As used in this document, the terms "comprise," "comprising," "include," "including," "contain," "containing," "having," "having," and all variations thereof are to be construed in an inclusive (i.e., non-exclusive) sense, such that the process, method, device, apparatus, or system described herein is not limited to the features, parts, elements, or steps recited, but may include other elements, features, parts, or steps not expressly listed or pertinent to the process, method, device, or apparatus. Furthermore, the terms "a" and "an" as used herein are to be construed to mean one or more, unless expressly stated otherwise. In addition, the terms "first," "second," "third," etc.used merely as designations and are not intended to impose numerical requirements on their objects or to establish a particular order of importance. LIST OF REFERENCE SYMBOLS 1 first vehicle 2 second vehicle 3 third vehicle 4 fourth vehicle A1 first wireless communication unit A2 second wireless communication unit AI module for artificial intelligence bd braking distance BDCM braking distance calculation module CD1 first image capture device CDlx further first image capture device CD2 second image capture device CD2x additional second image capture device CMS1 first image acquisition system CMS2 second image acquisition system CMSx additional image acquisition system DCM Distance Comparison Module dOD object detection distance dODm Minimum object detection distance DPU1 first data processing unit DPU2 second data processing unit DM digital card ID1 first image data ID2 second image data ID1+ID2 combined image data IDCM image data combination module F1 first field of view F2 second field of view L1 first lane L2 second lane ODD1 object distance detection module SCEM Safety Criteria Evaluation Module SLDU Speed ​​Limit Detection Unit VM vehicle maneuvers VO visual obstructions VSDU Vehicle Speed ​​Detection Unit QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 10,501,015 B2

[0002] US 11,014,490 B2

[0008] US 2019 / 318178 A1

[0009] Cited non-patent literature

[0000] Koch, T., Liebel, L., Fraundorfer, F., Körner, M. (2019). Evaluation of CNN-Based Single-Image Depth Estimation Methods. In: Leal-Taixé, L., Roth, S. (eds) Computer Vision - ECCV 2018 Workshops. ECCV 2018. Lecture Notes in Computer Science(), vol 11131. Springer, Cham., https: / / doi.org / 10.1007 / 978-3-030-11015-4_25

[0012]

Claims

[1] Vehicle safety and / or driver assistance method using a first image acquisition system (CSM1 - CSMx) with a first field of view (F1), the method comprising: Recording speed data; Generation of depth data for the first field of view (F1) and / or object detection distance data (dOD); Assessment, based on the recorded speed data, of whether a minimum depth criterion of the field of view (F1) and / or a safety criterion (dODm) of the minimum object distance (dODm) is met; in the event that the minimum depth criterion of the field of view (F1) and / or the safety criterion of the minimum object distance (dODm) is not met, combining the image data of the first image acquisition system with the image data of the second image acquisition system; Evaluation based on the combined image data as to whether a minimum depth criterion of the field of view (F1, F2) and / or a safety criterion of the minimum object distance (dODm) is met; and depending on the assessment of whether the criterion (dODm) is met or not met, output and / or display of safety data, whereby the safety data indicates whether the performance of a specific vehicle maneuver (VM) is safe or unsafe. [2] Vehicle safety and / or driver assistance method using image data from data-connected camera-based image acquisition systems mounted on at least one vehicle (1, 2, 3, 4), in particular according to claim 1, wherein the method comprises: Acquisition of image data with a first image acquisition system (CSM1); Applying an image analysis algorithm (AI) based on artificial intelligence and / or a neural network to the image data to generate depth data of the first field of view (F1) and / or object detection distance data (dOD); Assessment of whether a minimum depth of view criterion (F1) and / or a minimum object distance criterion (dODm) is met; in the event that the minimum depth criterion of the field of view (F1) and / or the criterion of the minimum object distance (dODm) is not met, receiving second image data via a data connection from at least one second image acquisition system (CSM2, CSMx) with a second field of view (F2); Combining image data from the first image acquisition system and image data from the second image acquisition system; and, on the basis of the combined image data, automatically assessing whether the performance of a particular vehicle maneuver (VM) is safe or unsafe, in particular using a software algorithm for assessing safety criteria, in particular using an algorithm based on artificial intelligence and / or a neural network for assessing safety criteria. [3] Method according to one of the preceding claims, wherein an image evaluation algorithm (AI) based on artificial intelligence and / or neural network comprises a machine learning model in the form of a neural network, in particular in the form of a transversal neural network and / or in the form of a convolutional neural network, correlated with a generative pre-trained transformer. [4] Method according to one of the preceding claims, further comprising Detecting speed limit data; correlating the detected speed limit data with the detected speed data of the first vehicle (1); Derivation of safety criteria data (bd) based on the correlated speed data; and Correlating the safety criteria data (bd) with the field of view depth data (F1) and / or the object distance data (dOD); in particular comprising the following steps: capturing speed limit data on the basis of further image data captured by an image capturing device (CD1, CD2) and / or on the basis of digital map data (DM), in particular on the basis of further image data which were captured by an image capture device (CD1, CD2) contained in the first vehicle (1) and / or on the basis of digital map data (DM) which are stored in a data storage unit of the first vehicle (1). [5] A method according to any one of the preceding claims, further comprising: Acquiring first image data from a first image capture device (CD1) or from a plurality of first image capture devices (CD1, CD1x) included in the first image capture system (CMS1); and in the event that the minimum depth criterion of the field of view (F1) and / or the safety criterion of the minimum object distance (dODm) is not met, capturing second image data from a second image capturing device (CD2) or from a plurality of second image capturing devices (CD2, CD2x) included in the second image capturing system (CMS2); in particular comprising the following steps: receiving in the first vehicle (1) in a first wireless communication unit (A1) contained in the first vehicle (1) image data of the second image acquisition system (CMS2) transmitted from a second wireless communication unit (A4) contained in a second vehicle. [6] Method according to one of the preceding claims, where the safety criteria data (bd) derived from the correlated speed data are data on the minimum safety braking distance, in particular data representing a minimum safety braking distance between a first vehicle (1) on a first lane (L1) and another vehicle (3) on the same lane (L1) or on another lane (L2, L3). [7] Method according to one of the preceding claims, wherein the method and / or the data are further suitable for avoiding a collision while a first vehicle (1) is performing a vehicle maneuver (VM) to change from a first lane (L1) to another lane (L2, L3). [8] Vehicle safety and / or driver assistance method using a first rearview image capture system with a first field of view (F1), in particular according to one of claims 1 to 7 or with one of the features of one of claims 1 to 7, the method comprising: Acquiring image data using the first rearview image acquisition system; Recording the speed data of the first vehicle (1); Applying a field of view depth detection algorithm (F1) and / or an object detection distance algorithm (dOD) to the image data to generate first field of view depth data (F1) and / or object detection distance data (dOD); Correlating the speed data of the first vehicle (1) with field of view depth data (F1) and / or the object distance data (dOD); Assessing, based on this correlation, whether a minimum depth criterion of the field of view (F1) and / or a safety criterion of the minimum object distance (dODm) is met; in the event that the minimum field of view depth criterion en (F1) and / or the minimum object distance safety criterion (dODm) is not met, receiving second image data using a second rearview image acquisition system with a second field of view (F2); Combining image data from the first rearview image acquisition system and image data from the second rearview image acquisition system; Assessment based on the combined image data, whether a minimum depth criterion of the field of view (F1, F2) and / or whether a safety criterion of the minimum object distance (dODm) is met; and depending on the assessment of whether the criterion (dODm) is met or not met, output and / or display of safety data, whereby the safety data indicates whether the performance of a specific vehicle maneuver (VM) is safe or unsafe. [9] Method according to one of the preceding claims, wherein the at least one image acquisition system is a camera monitoring system (CMS1, CMS2), in particular a rearview camera monitoring system (CMS1, CMS2), and wherein the image data are image data of the camera monitoring system. [10] Vehicle safety and / or driver assistance arrangement, in particular suitable for carrying out a method according to one of claims 1 to 9, wherein the arrangement comprises: a first image acquisition system having a first field of view (F1); at least one speed data acquisition unit (SLDU, VSDU); an object distance detection module (ODD1); a safety criteria data evaluation module (SCEM); a wireless communication unit (A1) suitable for receiving image data; a combination module image data combination module (IDCM) configured to combine image data of the first image acquisition system and at least one second image acquisition system; and a safety data output interface (SDOI) that is in data communication with the safety criteria data evaluation unit (SCEM). [11] Vehicle safety and / or driver assistance arrangement, in particular according to claim 10, in particular suitable for carrying out a method according to one of claims 1 to 9, wherein the arrangement comprises: a first rearview image acquisition system having a first field of view (F1); a first vehicle speed data acquisition unit (VSDU); a first speed limit data acquisition unit (SLDU); an object distance detection module (ODD1) which includes an artificial intelligence (AI) module; a safety criteria data evaluation module (SCEM); a wireless communication unit (A1) suitable for receiving image data; an image data combination module (IDCM) configured to combine image data from the first rearview image acquisition system and at least one second rearview image acquisition system; and a safety data output interface (SDOI) that is in data communication with the safety criteria data evaluation unit (SCEM). [12] Vehicle safety and / or driver assistance arrangement according to claim 10 or 11, wherein the first image capture system comprises at least one first image capture device (CD1, CD1x). [13] Vehicle safety and / or driver assistance arrangement according to one of claims 10 to 12, further comprising a speed limit data acquisition unit (SLDU) arranged to acquire speed limit data on the basis of image data acquired by an image acquisition system (CMSx) and / or on the basis of digital map data (DM). [14] Vehicle safety and / or driver assistance arrangement according to one of claims 10 to 13, wherein the at least one image acquisition system is a camera monitoring system (CMS1, CMS2), in particular a rearview camera monitoring system (CMS1, CMS2). [15] A data processing program product suitable for carrying out a vehicle safety and / or driver assistance method, comprising instructions which, when the program is executed by a data processing unit (DPU1, DPU2), cause the data processing unit (DPU1, DPU2) to carry out program steps which are suitable for carrying out a method according to one of claims 1 to 9, in particular in an arrangement according to one of claims 10 to 14, in particular a data processing program product suitable for carrying out a vehicle safety and / or driver assistance method using a first image acquisition system, in particular using a rearview image acquisition system (CMS1), with a first field of view (F1), comprises instructions which, when the program is executed by a data processing unit (DPU1, DPU2), cause the data processing unit (DPU1, DPU2) to carry out the following program steps,in particular according to a method according to one of claims 1 to 9:, a first program step (101) for reading first image data from the first image acquisition system; a second program step (102) of reading object distance data from an object distance detection module (ODD1, ODD2); a third program step (103) for reading speed data from a speed data acquisition unit (SLDU, VSDU); a fourth program step (104) in which it is assessed whether a minimum safety criterion (bd, dODm) is met; a fifth program step (105) for reading second image data from a first communication unit (A1), in particular second image data transmitted by a second communication unit (A2) which is in data communication with a second image acquisition system, if the minimum security criteria (bd, dODm) are not met; a sixth program step (106) for combining the image data of the first image acquisition system with the image data of the second image acquisition system; and a seventh program step (107) in which, on the basis of the combined image data, it is evaluated whether a minimum safety criterion (bd, dODm) is met; and an eighth program step (108) for outputting and / or displaying safety data, depending on the assessment of whether the criteria (bd, dODm) are met or not met.

Images