Method for localizing cameras, a camera system, and a vehicle with a camera system

Description

[0001] The invention relates to a method for locating cameras as well as a camera system and a vehicle with a camera system.

[0002] Nowadays, driver assistance systems are indispensable in motor vehicles. Driver assistance systems have functions that support the driver while driving. This is achieved primarily by the system detecting or recording the surroundings, processing the detected information, and then performing appropriate maneuvers or clearly presenting the surrounding information to the driver. Cameras are used in particular to detect the surroundings. For example, front cameras are well known, which are located near the windshield and capture the area in front of the vehicle. Rear-facing cameras are also state-of-the-art. Surround-view systems are also becoming increasingly common.Surround-view systems typically have four cameras, typically designed as fisheye cameras. One camera is located at the front of the grille, one at the rear, and one at the base of each of the vehicle's side mirrors. Each camera can have a field of view of more than 180°, allowing the system to capture the entire vehicle's surroundings in a 360° radius. The images from the cameras of common surround-view systems are fused, and it is important that it is precisely clear which camera each image is coming from. Coded cameras are used for this purpose. This involves additional effort and thus increased costs and time. Another option for detecting the camera position is to connect the cameras directly to a fusion unit with specially coded ports. The disadvantage of this approach, however, is that errors can easily occur when plugging in the cables and go unnoticed.Furthermore, communication switches are increasingly being used, which means that the position of the cameras can no longer be determined based on the connection, since the data streams in particular are aggregated on a common physical link.

[0003] EP 2 530 647 A1 describes a method for calibrating a surround-view system of a vehicle comprising four wide-angle cameras arranged at different locations on the vehicle and an image processing unit connected to the cameras. The image processing unit determines the relative position and orientation between the cameras based on captured images of several specific calibration objects (in particular, checkerboard-like markings) positioned within an overlapping area of ​​the cameras' fields of view (FOVs). Thus, image pairs captured by cameras with overlapping fields of view are evaluated.For this purpose, an image of the calibration objects is captured with a first camera and a further image of the calibration objects is captured with a second camera, wherein the calibration objects are positioned in an overlapping area of ​​a field of view of the first camera and a field of view of the second camera, and wherein the calibration objects are in the same position relative to the vehicle during the capture of these images. A position and orientation of the first camera relative to the calibration objects are then determined based on the captured image, and a position and orientation of the second camera relative to these calibration objects are determined based on the further captured image. Based on this, a relative position and orientation between the second camera and the first camera is subsequently determined. This is carried out for all pairs of cameras with overlapping fields of view.Based on the relative positions and orientations of the cameras thus determined and a determined absolute position and orientation of any of the four cameras, the absolute positions and orientations of the cameras of the surround view system are then determined.

[0004] US 2016 / 284087 A1 discloses a method that uses the movement of a vehicle to determine the orientation of a vehicle's camera system relative to the vehicle. For this purpose, feature points on a road surface are identified, and the positions and angles of the feature points relative to a camera coordinate system are then determined using a structure-from-motion algorithm. The determined feature points are used as a first condition for determining the extrinsic camera parameters. Edges of the vehicle depicted in the camera images are also used as a second condition.

[0005] Based on this, it is now the object of the invention to provide a method which enables a simple and less error-prone determination of the position of individual cameras of a surround-view system.

[0006] This object is achieved by a method having the features of independent patent claim 1. Preferred embodiments are the subject of the dependent claims. A camera system and a vehicle are the subject of the independent patent claims.

[0007] According to a first aspect, the present invention relates to a method for locating cameras, in particular cameras of a surround view system, comprising the steps: Activating the cameras, capturing images and / or videos by the cameras, analyzing data streams, creating a linked list of the cameras, identifying at least one absolute camera position and / or one camera, and deriving the absolute positions of the remaining cameras.

[0008] By analyzing the data streams, a linked list of cameras is obtained. In particular, this reveals the cameras' interdependence. This allows the relative position of each camera to be defined. This essentially provides information about which cameras are located next to each other. In principle, each camera has a successor and a predecessor. The exact position of the individual cameras, especially the absolute position, cannot yet be determined by this step. However, if the absolute position of a single camera is known, the absolute positions of the remaining cameras can also be determined based on the linked list. Advantageously, exactly one single absolute camera position is determined.

[0009] The core of the invention lies in the provision of a two-stage method, wherein, in a first step, a linked list of the cameras is created. This allows the relative position of each camera to be clearly defined. In a further step, at least one absolute position of a camera is identified. Based on the absolute camera position and the linked list, the absolute positions of the other cameras can then be derived.

[0010] A key advantage of the method according to the invention is that it can dynamically detect the position of each camera without the use of additional hardware. This is not only important for new installations but also reduces the costs of replacing a camera. The method according to the invention also has the advantage that generic cameras can be used. Furthermore, the method can prevent coding and programming errors. Furthermore, common hardware does not need to be modified, allowing the method to be integrated into existing networks.

[0011] In particular, the method is used to locate cameras on a vehicle. The method can thus preferably be used to determine the exact location of the cameras, for example, whether the camera is located in the front, rear, or side of the vehicle. With the help of the invention, coded cameras are not required; generic cameras can preferably be used, and the cameras do not need to be assigned fixed positions.

[0012] For the purposes of the invention, data streams can be understood as both images and video frames. The data streams of cameras, especially Ethernet cameras, can be easily identified as such, since special protocols for embedding videos are preferably used.

[0013] According to the invention, it is sufficient if not all cameras are activated. However, it is preferable for all cameras to be activated. It is advantageous if every camera records images and / or videos. However, this is not absolutely necessary. It is generally sufficient if only some cameras record images or videos.

[0014] The analysis of the data streams is preferably carried out in a fusion unit. For this purpose, the camera data streams, in particular, are sent to the fusion unit. It is advantageous if sender addresses are extracted from the received data streams. This allows the data to be assigned to the individual cameras.

[0015] In a preferred embodiment, time synchronization is performed. This time synchronization allows precise conclusions to be drawn about the recording time of the individual cameras. This allows, in particular, data streams from different cameras recorded at the same time to be compared with each other. In particular, this time synchronization allows video frames or images to be assigned to one another, even if the data take different paths within the vehicle network. Time synchronization preferably takes place after the cameras are activated and / or before images are captured.

[0016] For time synchronization, common methods are used. These methods are based primarily on standards such as Autosar 4.2.2, gPTP, IEEE 802.1AS, and IEEE802.1AS-rev, which emerged from the IEEE Audio / Video Bridging Task Group.

[0017] It is advantageous if the analysis of the data streams includes a comparison of the data streams. In particular, the analysis includes a comparison of the data streams with the same time unit. When comparing the data streams, the focus is on looking for similarities, preferably similarities in peripheral areas. Since neighboring cameras preferably have an overlapping area, i.e. partially cover the same area, they have similarities in the data streams, at least in peripheral areas. If similarities are identified when comparing the data streams of two cameras, it can be concluded that these cameras are neighbors. These cameras are then listed one after the other in the linked list.

[0018] It is also possible, in principle, to examine the quality of the images or video frames. If the quality is sufficient, the respective sequences, especially those images or video frames with the same time unit, are compared. If the quality of the images is insufficient, a backup procedure is initiated.

[0019] The preferred method is to identify the number of cameras that are active and / or recording images or videos.

[0020] According to one embodiment of the present invention, the fusion unit can store the images and / or video frames. The fusion unit preferably stores the images and / or video frames in a buffer. In particular, data streams with the same time unit are selected and then stored. If the system has four cameras, for example, the images or video frames are preferably selected with the same time unit and then stored. Consequently, four images are then available in the buffer.

[0021] To keep the data volume as manageable as possible, it is preferable to record and compare only a few sequences or a snapshot. This is made possible, in particular, by time synchronization.

[0022] In a preferred embodiment of the invention, to identify at least one absolute camera position, a check is carried out to determine whether a known absolute position exists for at least one camera. This can be the case, in particular, if a camera has been replaced or a camera is specially coded. If a known absolute position exists, the positions of the other cameras can be easily derived using the linked list.

[0023] To identify the absolute positions of the cameras, the images or video frames can also be analyzed with regard to an operating mode and / or driving mode and / or the detection of objects. A gyro, a steering angle, a directly selected mode of travel such as forward or reverse gear, or an indirect mode of travel such as on a conveyor belt, a truck, or a ship can be selected as the driving mode. Identification is carried out in particular by first activating or starting the cameras and then analyzing the image content. A driving mode or operating mode is also selected that changes the position of the vehicle. Changes in the image content are then preferentially identified. Based on these changes, the absolute position of the camera can then be derived.

[0024] If the camera system has four cameras, then with the method according to the invention it is in principle sufficient if a neighbor is found for only two cameras. For example, if camera A is the neighbor of camera B and camera C is the neighbor of camera D, then a linked list can still be created even though two links are missing. For example, if camera A is the left neighbor of camera B and camera D is the right neighbor of camera C, then the list looks like this: ABCD. This can be clearly deduced. The method therefore only needs to recognize two neighborhoods in which at least 3 nodes are involved. The method therefore works even if a camera is deactivated, not mounted, or has an error.

[0025] According to a second aspect, the invention relates to a camera system comprising a plurality of cameras for capturing a vehicle's surroundings and a fusion unit. The fusion unit is configured to analyze received data streams from the cameras, thereby creating a linked list of the cameras, to identify at least one absolute camera position, and to derive the absolute positions of the remaining cameras based on the at least one absolute camera position and the linked list. The camera system is particularly suitable for implementing the method according to the invention.

[0026] The cameras, in particular, have an aperture angle of at least 180°. It is advantageous if the cameras are fisheye cameras. The cameras of the camera system preferably have different detection ranges, with neighboring cameras, in particular, having an overlapping range. Advantageously, the cameras are generic cameras, i.e., generally applicable cameras. The cameras can all be designed identically.

[0027] The camera system ideally comprises four cameras. It is advantageous to use a surround-view camera system.

[0028] According to a third aspect, the invention relates to a vehicle with a camera system according to the invention. The cameras are preferably arranged on different sides of the vehicle.

[0029] Further advantageous embodiments are shown in the drawings. Fig. 1: Schematic representation of a flowchart of a method according to the invention for locating cameras in one embodiment; Fig. 2: Schematic representation of a flowchart for creating a linked list in one embodiment; Fig. 3: Schematic representation of a flowchart for determining absolute camera positions in one embodiment; Fig. 4: Schematic representation of a camera system according to the invention in one embodiment.

[0030] Figure 1shows a schematic representation of a flowchart of a method according to the invention for locating cameras 12A, 12B, 12C, 12D in one embodiment. The cameras 12A, 12B, 12C, 12D are preferably cameras arranged on a vehicle, wherein the cameras are in particular assigned to a surround-view system. The camera 12A is preferably arranged at the front on the radiator grille, the camera 12C at the rear, and the cameras 12B and 12D are each arranged in the region of a side mirror of the vehicle. The adjacent cameras 12A, 12B, 12C, 12D at least partially have an overlap region 16AB, 16BC, 16CD, 16DA.

[0031] According to the invention, the cameras 12A, 12B, 12C, 12D are activated in a first step (S1). It is advantageous if time synchronization is carried out. Through time synchronization, precise conclusions can be drawn about the recording time of the individual cameras 12A, 12B, 12C, 12D. This then allows data streams from the various cameras 12A, 12B, 12C, 12D that were recorded at the same time to be compared. Through time synchronization, in particular, video frames or images can be assigned to one another, even if they have to travel different distances in the vehicle network. Time synchronization takes place in particular before the images or videos are recorded. In a second step, the cameras 12A, 12B, 12C, 12D record images or videos of the vehicle's surroundings (S2).

[0032] In a third step, data streams are analyzed (S3). The analysis preferably takes place in a fusion unit 14. It is advantageous if the analysis of the data streams includes a comparison of the data streams, in particular the data streams with the same time unit. When comparing the data streams, commonalities in peripheral areas are preferably sought. Since neighboring cameras preferably have an overlapping area 16AB, 16BC, 16CD, 16DA, i.e. partially cover the same area, the data streams have commonalities at least in the peripheral areas. If commonalities are identified when comparing data streams from two cameras, it can be concluded that these cameras are neighbors.

[0033] Overall, analyzing the data streams yields a linked list of cameras (S4). This reveals the cameras' interdependencies. This allows the relative position of each camera to be defined. Essentially, this provides information about which cameras are located next to each other. Each camera essentially has a successor and a predecessor.

[0034] In a further step, at least one absolute camera position and / or a camera is identified (S5). When identifying the position and / or the camera, a check can be made to determine whether a known absolute position already exists for at least one camera. This can be the case, in particular, if a camera has been replaced or if a camera is specially coded. Furthermore, to identify the absolute positions of the cameras, an analysis of the image or video frame can be performed with respect to an operating mode and / or driving mode and / or the detection of objects.

[0035] If the absolute position of a single camera is known, the absolute positions of the remaining cameras can be derived from it. The linked list is used for this purpose.

[0036] Figure 2shows a schematic representation of a flow diagram for creating a linked list in one embodiment. In a first step, for example, the video stream or data stream from camera 12A is analyzed. The camera can be selected randomly or according to a predetermined pattern. For example, one can start with the camera with the smallest sender address. Then one uses a data stream from another camera 12B, 12C, 12D. The data streams are compared with each other, looking for overlaps or similarities. Since neighboring cameras usually have areas of overlap, the recorded images or videos from neighboring cameras have similarities or overlaps in their edge areas. Therefore, for example, one compares the right image section from camera 12A with the left image section from camera 12B.If an overlap is detected here, this allows the conclusion that these are adjacent cameras, and camera 12B can be placed behind camera 12A in the linked list. The next step is to compare, for example, the left image section of camera 12A with a right image section of camera 12D. If overlaps occur here, camera 12D can be placed before camera 12A in the linked list. If the camera system has four cameras, camera 12C can now simply be placed between cameras 12D and 12B without examining any images or data streams from this camera. Overall, a linked list of cameras has been created, whereby the relative positions of the cameras are clearly defined.

[0037] If gaps arise during the analysis, the search can be repeated with a different camera and the data sets already found can be supplemented with the new ones.

[0038] Figure 3 shows a schematic representation of a flow chart for determining absolute camera positions in one embodiment. For this purpose, the linked list, i.e., the camera list with the results of the relative assignment, is preferably loaded in a first step. Then, a check is carried out to determine whether there is a camera for which an absolute position is already known. This can be the case, for example, if a camera is specially coded. If this is the case, all other positions can be derived from the known position and using the linked list.

[0039] If no absolute camera position is known, the cameras are started, either individually or all together. A driving or operating mode is then selected that changes the vehicle's position. If the vehicle's position changes, this generally also results in changes in the camera data streams. In a subsequent step, the image content is analyzed in relation to the selected mode. If the vehicle is moving, for example, on a production line, during transport, or at a customer site, it is very easy to distinguish between the front and rear cameras. The image analysis allows conclusions to be drawn about the camera position, which then makes it possible to assign all cameras to fixed positions.

[0040] Figure 4shows a schematic representation of a camera system 10 according to the invention in one embodiment. The camera system 10 comprises a plurality of cameras 12A, 12B, 12C, 12D for capturing a vehicle's surroundings. The camera system 10 further comprises a fusion unit 14. The fusion unit 14 is designed to analyze received data streams from the cameras 12A, 12B, 12C, 12D, thereby creating a linked list of the cameras. Furthermore, the fusion unit 14 is designed to identify at least one absolute camera position and to derive the absolute positions of the remaining cameras 12A, 12B, 12C, 12D based on the at least one absolute camera position and the linked list.

[0041] The cameras 12A, 12B, 12C, 12D, in particular, have an aperture angle of at least 180°. The cameras of the camera system preferably have different detection ranges, with neighboring cameras, in particular, having an overlapping range 16AB, 16BC, 16CD, 16DA. Advantageously, the cameras 12A, 12B, 12C, 12D are generic cameras.

[0042] Advantageously, the camera system 10 is a surround-view camera system, wherein the system comprises four cameras 12A, 12B, 12C, 12D.

[0043] The invention is defined by the independent claims. The dependent claims define advantageous embodiments.

Claims

1. Method for locating cameras (12A, 12B, 12C, 12D), in particular cameras of a surround-view system, comprising the following steps: - activating the cameras (12A, 12B, 12C, 12D) (S1), - recording images and / or videos by means of the cameras (12A, 12B, 12C, 12D) (S2), - analysis of data streams (S3), - creating a concatenated list of the cameras (12A, 12B, 12C, 12D) (S4), - identifying at least one absolute camera position and / or one camera (12A, 12B, 12C, 12D) (S5), - deriving the absolute positions of the remaining cameras (12A, 12B, 12C, 12D) (S6), characterized in that a time synchronisation is carried out, and in that the analysis of the data streams comprises a comparison of the data streams with the same time unit.

2. Method according to the preceding claim, characterized in that the comparison comprises a search for commonalities in the data streams, in particular in edge regions.

3. Method according to either of the preceding claims, characterized in that the number of cameras (12A, 12B, 12C, 12D) which are active and / or record images or videos is identified.

4. Method according to any of the preceding claims, characterized in that the images and / or video frames are stored.

5. Method according to the preceding claim, characterized in that data streams with the same time unit are stored together.

6. Method according to any of the preceding claims, characterized in that for identifying at least one absolute camera position, a check is made to ascertain whether a known absolute position is present for at least one camera (12A, 12B, 12C, 12D).

7. Method according to any of the preceding claims, characterized in that for identifying the absolute position, an analysis of the image or the video frame with respect to an operating mode or driving mode and / or with respect to the recognition of objects is carried out.

8. Camera system (10), in particular for carrying out a method according to any of the preceding claims, comprising a plurality of cameras (12A, 12B, 12C, 12D) for capturing a vehicle environment and a fusion unit (14), wherein the fusion unit is configured in such a way that it analyses received data streams from the cameras (12A, 12B, 12C, 12D), thereby creating a concatenated list of the cameras (12A, 12B, 12C, 12D), that it identifies at least one absolute camera position and that it derives the absolute positions of the remaining cameras (12A, 12B, 12C, 12D) on the basis of the at least one absolute camera position and the concatenated list, characterized in that the fusion unit is furthermore configured in such a way that it performs a comparison of the data streams with the same time unit in the context of the analysis of the data streams.

9. Camera system (10) according to Claim 8, characterized in that it is a surround-view camera system and the cameras (12A, 12B, 12C, 12D) have an aperture angle of at least 180° and adjacent cameras (12A, 12B, 12C, 12D) have an overlap region (16AB, 16BC, 16CD, 16DA).

10. Vehicle (18) comprising a camera system (10) according to either of Claims 8 and 9.