COMPUTER-IMPLEMENTED METHOD FOR DETERMINING A POSITION VECTOR

Abstract

A computer-implemented method 126 for determining a position vector 124 between an eye 110 and an object 104, comprising a first detection step 128 in which an object position vector 116 relative to the first 3D camera 106 is detected by means of a first 3D camera 106; a second detection step 130 in which an eye position vector 114 relative to the second 3D camera 152 is detected by means of a second 3D camera 152; a provision step 132 in which a predetermined relation vector 159, which indicates the position of the first 3D camera 106 relative to the second 3D camera 152, is provided; and a calculation step 134 in which the position vector 124 from the eye 110 to the object 104 is calculated based on the object position vector 116, the eye position vector 114 and the relation vector 159.

Description

Technical FieldThe present invention relates to the field of image processing. In particular, the present invention relates to a computer-implemented method for determining a position vector between an eye and an object. The present invention further relates to a computer-implemented method for the environment-synchronous display of image data on a display. The present invention furthermore relates to a computer program product, a computer-readable storage medium, a data carrier signal, a device for data processing and a vehicle.Background ArtImproving safety measures for car drivers is an important goal for all vehicle manufacturers. Unsafe traffic situations can arise due to non-transparent structures of the vehicle body, which they can conceal traffic-relevant events for a driver. Such non-transparent structures are usually parts of the vehicle body which, in the event of an accident, ensure the mechanical rigidity required for the safety of the occupants. Examples of such non-transparent structures are the A, B and C pillars and the dashboard or the trunk. The non-transparent structures interfere with the field of vision of the driver, resulting in unpredictable areas (so-called blind spots). Traffic safety-relevant events or objects such as a pedestrian who wishes to cross the road, a cyclist or an overtaking car are therefore covered and possibly not recognized in good time by the rider. The opposing requirements for a high rigidity of the vehicle body on the one hand and a large field of view for the driver on the other hand generally require a compromise.As solutions, transparent A-pillars, engine hoods or tailgates have been proposed. These are displays, for example in the form of LCD or OLED screens, which are placed inside the vehicle between the non-transparent structure and the driver and display images which originate from a camera which is arranged outside the vehicle and whose field of view captures the non-visible region caused by the structure. Such an arrangement creates the inclusion that the structure is transparent.For example, EP 1 878 618 B1 discloses an assistance unit for capturing an image of a driver-invisible area that arises due to the presence of a pillar of a vehicle.The solutions disclosed in the prior art have the disadvantage that the images are not displayed in an environmentally synchronous manner. In this context, "environment-synchronous" is to be understood to mean that the image displayed on the display should be adapted such that it smoothly enters the environment from the perspective of the driver. In other words, the viewable area should appear to the driver as being closely associated with the image of the invisible area.It is therefore the object of the invention to provide a computer-implemented method for determining a position vector between an eye and an object and a computer-implemented method for the environment-synchronous display of image data on a display. Further objects of the invention are to provide a computer program product, a computer-readable storage medium, a data carrier signal, a device for data processing and a vehicle.Disclosure of the InventionThe object is achieved according to the invention by a computer-implemented method and a computer program product, a computer-readable storage medium, a data carrier signal, a device for data processing and by a vehicle according to the respective main claims. Advantageous embodiments can be gathered from the dependent claims.According to a first aspect of the invention, a computer-implemented method for determining a position vector between an eye and an object has a first step in which an object position vector is detected relative to the first 3D camera by means of a first 3D camera.3D cameras are systems for distance determination. For example, such a system can be equipped with a camera for image acquisition and a distance sensor such as radar or lidar. However, this can also be a pure radar system. Frequently, 3D cameras also contain at least one arithmetic unit and / or memory in order to carry out tasks of image processing, for example object recognition and classification.If the first 3D camera generates image data, objects can be detected in them by object detection. These may be trees, people, vehicles or the like, for example. Objects can also be detected from radar data.Some 3D cameras can also capture depth information in addition to image data. For example, a 3D camera can then add depth information to each pixel. Associated with object detection, the first 3D camera may generate position vectors for objects in their field of view. These position vectors are typically to be understood relative to the first 3D camera.From the detected objects, one is selected as a basis for the method described here. It can be, for example, the object with the smallest distance.The object is advantageously struck on the basis of a safety-relevant classification of the objects. If the method is used in a vehicle, for example, and if a tree and a person are identified in the image data during a turn process, then the person can be classified as more relevant for safety, for example, with respect to the current driving situation.In a further step of the method, an eye position vector relative to the second 3D camera is captured by means of a second 3D camera. The second 3D camera can be designed, for example, in the interior of a vehicle as a gaze detector (eye tracker) and thus detect the eyes of the driver. The second 3D camera can also capture depth information and thus provide a position vector of one or both eyes, for example that of a driver. These position vectors are typically to be understood relative to the second 3D camera.In a further step of the method, a predetermined relation vector is provided which indicates the position of the first 3D camera relative to the second 3D camera. This predetermined relation vector is a known, static vector which is made available, for example, by means of a memory unit. The relation vector can also be defined, for example, via a vehicle-fixed coordinate system. In other words, the positions of the first 3D camera and the second 3D camera are defined relative to a vehicle-mounted coordinate system. The position relative to one another can thus also be calculated.In a further step of the method, the position vector between the eye and the object is calculated on the basis of the eye position vector, the object position vector and the relation vector. The length of the position vector is decisive if it is desired to display the object on a screen in an environment-synchronous manner, i.e. if the object is made to appear to an observer as if it is closely connected to the environment of the screen. Since both eye and object have dynamic positions, a static displacement vector is not sufficient to calculate the exact distance vector between eye and object.According to a second aspect of the invention, a computer-implemented method for the environment-synchronous representation of image data on a display comprises a first step, in which the position vector between an eye and an object is determined as described above.In a further step of the method, image data is captured by the first 3D camera as described above.In a further step of the method, intersection points between the position vector and a projection surface of the display are determined. The projection area of the display may correspond to the area of the active screen of the display, for example. Mathematical methods for determining intersection points between vectors and surfaces are known to the person skilled in the art.In a further step of the method, partial image data is generated from the image data at least partially on the basis of the position vector. The sub-image data is generated such that it is perceived in an environmentally synchronous manner from the perspective of the eye when it is displayed on the display. In other words, the image data is to be adapted such that the partial image data to be displayed on the display is seamlessly inserted into the environment from the perspective of the observer to whom the eye belongs. Knowing the distance from the eye to the object is crucial to such an effect. During the generation, the first image data can be cropped, magnified, reduced or otherwise geometrically deformed, for example. This step is only carried out if the preceding step has revealed that at least one intersection point exists, because otherwise the display is not in the eye-to-object path of view.The sub-image data is generated such that it is perceived by the driver in an environmentally synchronous manner when it is displayed on the display.In a final step of the method, the partial image data are displayed on the display.Using the method as described above, the image can be displayed in an improved manner on a transparent A-pillar or transparent engine hood.According to a third aspect of the invention, a computer program product comprises instructions which, when the program is executed by a computer, cause the computer to execute a computer-implemented method as described above.According to a fourth aspect of the invention, a computer readable storage medium comprises instructions which, when executed by a computer, cause the computer to execute a computer implemented method as described above.According to a fifth aspect of the invention, a data carrier signal transmits the computer program product as described above.According to a sixth aspect of the invention, a device for data processing for the environment-synchronous display of image data on a display has an evaluation unit. This device is designed such that it can carry out the steps of a computer-implemented method as described above.The evaluation unit can be designed, for example, with a processor, a computer-readable storage medium and data inputs and outputs. The storage medium may store, for example, a computer program product whose instructions cause the processor to execute the steps of a computer-implemented method as described above. Furthermore, necessary data, such as the relation vector, can be stored on the computer-readable storage medium.The device also has a display which is communicatively connected to the evaluation unit. In other words, the evaluation unit can generate an image to be displayed and transmit it to the display by means of a suitable data output.Advantageously, the display is arranged on a structure of a vehicle which blocks the view of a driver of the vehicle to an environment outside the vehicle.Furthermore, the device according to the sixth aspect of the invention has a first 3D camera, which is communicatively connected to the evaluation unit. The first 3D camera provides the image data which is to be displayed on the display in an environmentally synchronous manner.In an advantageous embodiment, the first 3D camera at least partially captures the environment of the vehicle in the image data, which environment is not visible to the driver through the structure of the vehicle.In a particularly advantageous embodiment, the structure of the vehicle is the A-pillar, the B-pillar, the C-pillar or the dashboard.In addition, the device according to the sixth aspect of the invention has a second 3D camera, which is communicatively connected to the evaluation unit.Advantageously, the second 3D camera is arranged in the interior of the vehicle in such a way that it can capture at least one eye of the driver. In this way, the eye position vector required for the method to be carried out with the device can be easily determined.In a preferred embodiment, the first 3D camera and / or the second 3D camera is configured with a laser, a radar, a lidar, at least two cameras or an ultrasonic sensor. Such sensors make it possible to provide depth information in addition to the image data, for example via a runtime evaluation or a disparity. Positions of objects identified in the image data (which also include the eyes of the driver) can thus be determined.According to a fifth aspect of the invention, a vehicle has a data processing device as described above.Summary of the FiguresThe invention is explained in more detail below with reference to exemplary embodiments with the aid of figures. The figures show: FIG. 1 is a schematic overview of the problem underlying the invention; FIG. 2 : shows a flow diagram of a computer-implemented method for determining a position vector; FIG. 3 : shows a flow diagram of a computer-implemented method for the environment-synchronous representation of image data on a display; FIG. 4 : a plan view of a vehicle for explaining an embodiment of the method from FIGS. 2 and 3 ; FIG. 5 : shows a device for data processing which can carry out the method from FIG. 3 ; and FIG. 6 : A vehicle with the device for data processing from FIG. 5.Detailed Description of the FiguresFIG. 1 shows a schematic overview of the problem on which the invention is based. In FIG. 1 a), a display 100 on a projection surface 102 shows an object 104 located behind the display 100 in the form of a tree. The display 100 is configured as a smartphone in FIG. 1, the screen surface of which forms the projection surface 102. The object 104 is shown on the display 100 so small that it does not insert itself into its environment in an environmentally synchronous manner. In other words, the image on the display 100 does not appear closely associated with the real environment.FIG. 1 b) shows the result of the non-environment-synchronous representation in FIG. 1 a). The display 100 has a first 3D camera 106, with which the object 104 behind the display 100 can be detected. The first 3D camera 106 has a first field of view 108 that completely captures the object 104 that is located at a first distance 116 from the first 3D camera 106. The first 3D camera 106 therefore captures the entire object 104, which consists of a first sub-object 118, a second sub-object 120 and a third sub-object 122.An eye 110 of an observer is located a second distance 114 from the display 100 and is looking at the projection surface 102 of the display 100. Thus, a second field of view 112 is defined, which is covered by the projection surface 102 and is therefore not visible to the eye 110. The portion of the object 104 that is not visible is the first sub-object 118. The second object part 120 and the third object part 122 are furthermore visible to the eye 110.If an unchanged image of the first 3D camera 106 is now displayed on the projection surface 102, the display of the second sub-object 120 of the third sub-object 122 has the result that the display does not appear to be environmentally synchronous (as is shown in FIG. 1 a)). If, in addition to the first distance 116, a second distance 124 is also known, which corresponds to the distance between the eye 110 and the object 104, then the part which leads to an environmentally synchronous representation on the projection surface 102 can be determined from the image data of the first 3D camera 106 by simple geometric calculations. In the example of FIG. 1, the image data of the first 3D camera 106 can be cropped, for example, such that the second sub-object 120 and the third sub-object 122 are cut away. This results in an environmentally synchronous representation of the image data with a corresponding magnification, as can be seen in FIG. 1 c).FIG. 2 shows a flow diagram of a computer-implemented method for determining a position vector 126.In a first detection step 128, an object position vector 116 is detected relative to the latter by means of a first 3D camera 106. This step is carried out continuously, as shown by the circular arrow in FIG. 2.In a second acquisition step 130, an eye position vector 114 relative to the second 3D camera 152 is acquired by means of a second 3D camera 152. This step is carried out continuously, as shown by the circular arrow in FIG. 2.In a providing step 132, a predetermined relation vector 159 is provided, wherein said relation vector indicates the position of the first 3D camera 106 relative to the second 3D camera 152.In a calculation step 134, a position vector 124 is calculated, which points from the eye 110 to the object 104. The calculation step 134 is carried out on the basis of the object position vector 116 from the first acquisition step 128, the eye position vector 114 from the second acquisition step 130, and the relation vector 159 from the provision step 132.FIG. 3 shows a flow diagram of a computer-implemented method for the environment-synchronous representation of image data 136 on a display 100.The method for the environment-synchronous representation of image data 136 builds on the method for determining a position vector 126. In a first step, the position vector 124 is therefore determined by means of the method for determining a position vector 126.In a third detection step 138, which is carried out continuously (illustrated in FIG. three as a circular arrow), the image data are detected by the first 3D camera 106. This is the same first 3D camera 106, which is also used to determine the position vector 124. By object recognition and classification, a position vector 124 can thus be assigned to an object 104.In a determination step 140, intersection points between the position vector 124 and a projection surface 102 of the display 100 are determined. This can be achieved using known methods of mathematics.In a decision step 142, it is checked whether at least one intersection point has been determined in the previous determination step 140. If this is not the case (f branch in FIG. 3 ), the method 136 is ended. Unchanged image data of the first 3D camera can also be displayed.If, however, at least one intersection point was determined in the determination step 140 (t branch in FIG. 3 ), then partial image data is subsequently generated from the image data in a generation step 144. This occurs at least partially based on the position vector 124 such that the sub-image data is perceived in an environmentally synchronous manner from the perspective of the eye 110 when it is displayed on the display 100. For example, the partial image data is generated by reduction, enlargement, displacement, geometric distortion and / or cropping of the image data.In a display step 146, the partial image data are displayed on the display 100.To explain the methods 126, 136 of FIGS. 2 and 3 in more detail, FIG. 4 shows a top view of a vehicle 148. A driver 150 is located in the vehicle 148.The vehicle 148 includes a first 3D camera 106 and a second 3D camera 152. Outside the vehicle 148 is an object 104 which is not completely visible to the driver 150 through the A-pillar of the vehicle 148, that is to say is located in a region which cannot be seen. A display 100 is disposed on the A-pillar in the interior of the vehicle 148. The display 100 has a projection surface 102 defined by the screen of the display 100. The screen of the display 100 and thus the projection surface 102 can be curved in order to adjoin the A-pillar.The first 3D camera 106 is arranged such that the region that is not visible to the driver 150 can be captured therewith, for example directly opposite the display 100 or, as in the example of FIG. 3, on an exterior mirror of the vehicle 148. The first 3D camera 106 captures image data of the invisible area. However, the first 3D camera 106 can also consist of a plurality of cameras, which monitor a relatively large environment of the vehicle 148 and do not specifically capture only the invisible area.Computer-assisted object recognition can be used to recognize a set of objects in the image data. For this purpose, the data can be evaluated directly by the first 3D camera or the raw data can first be transmitted to an evaluation unit 170. If a plurality of objects are detected in the image data, the object 104 can be selected from the set of objects on the basis of a safety-relevant classification of the objects. By means of a corresponding module, it is possible, for example, to calculate collision probabilities with further road users and objects in the environment of the vehicle 148, with the result that it is possible to select that object 104 for which the highest collision probability has been calculated.The first 3D camera 106 also acquires depth information relative to itself in relation to the image data. This can be realized, for example, with a laser, a radar, a lidar or an ultrasonic sensor. If the first 3D camera 106 has two cameras, depth information can also be calculated using a disparity in the respective image data. In the example of FIG. 4, the first 3D camera 106 provides an object position vector 116. A first distance 116 between the first 3D camera 106 and the object 104 may be calculated as the length of the object position vector 116.The second 3D camera 152 is disposed in the interior of the vehicle 148 so as to be able to capture the eyes 110 of the driver 150. Like the first 3D camera 106, the second 3D camera 152 can also provide a position of an eye 110 as eye position vector 114 by means of object detection and a corresponding sensor. The eye position vector 114 is defined relative to the second 3D camera 152.A vehicle-mounted coordinate system 154 may be freely selected, for example, such that the position of a particular component in the vehicle 148 defines the origin. The direction of travel can be defined, for example, as the abscissa, in order to thus form a Cartesian coordinate system. A first position vector 156 describes the position of the first 3D camera 106 relative to the vehicle-mounted coordinate system 154. A second position vector 158 describes the position of the second 3D camera 152 relative to the vehicle-mounted coordinate system 154. Since both the first position vector 156 and the second position vector 158 are known and invariable, a vehicle-eye position vector 160 and a vehicle-object position vector 162 can be calculated via them, which are now defined relative to the vehicle-mounted coordinate system 154.Knowing the vehicle eye position vector 160 and the vehicle object position vector 162, it is possible to determine the position vector 124 pointing from the eye 110 of the driver 150 to the object 104. However, it is not necessary to introduce a vehicle-fixed coordinate system 154 if a relation vector 159 is known which defines the position of the first 3D camera 106 relative to the second 3D camera 152.In the example of FIG. 4, the position vector 124 intersects the projection surface 102 of the display 100. The method 136 of FIG. 3 provides for the generation of partial image data for this case, which partial image data is perceived in an environmentally synchronous manner from the perspective of the eye 110 when it is displayed on the display 100. This is because only in this case are at least parts of the object 104 in the unpredictable area.FIG. 5 shows a data processing apparatus 164 that may perform the method 136 of FIG. 3. The device 164 includes a display 100, a first 3D camera 106, and a second 3D camera 152. Furthermore, the device 164 has an evaluation unit 170 which consists of a processor 166 and a computer-readable storage medium 168. All components of the device 164 are communicatively connected to one another.The computer readable storage medium 168 stores a computer program product in a programming language, instructions of which, when executed by the processor 166, execute the method 136 of FIG. 3. In addition, a relation vector 159 required for carrying out the method 136 from FIG. 3 is stored on the computer-readable storage medium 168.FIG. 6 shows a vehicle 148 with the data processing device 164 from FIG. 5.List of reference characters100 Display 102 Projection surface 104 Object 106 First 3D camera 108 First field of view 110 Eye 112 Second field of view 114 Second distance, eye position vector 116 First distance, object position vector 118 First sub-object 120 Second sub-object 122 Third sub-object 124 Distance eye-object, Position vector 126 Method for determining a position vector 128 First acquisition step 130 Second acquisition step 132 Provision step 134 Calculation step 136 Method for the environment-synchronous representation of image data 138 Third acquisition step 140 Determination step 142 Decision step 144 Generation step 146 Representation step 148 Vehicle 150 Driver 152 Second 3D camera 154 Vehicle-fixed coordinate system 156 First position vector 158 Second position vector 159 Relation vector 160 Vehicle eye position vector 162 Vehicle object position vector 164 Device 166 Processor 168 Storage medium 170 Evaluation unitReferences included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedEP 1 878 618 B1

[0004]

Claims

Computer-implemented method (126) for determining a position vector (124) between an eye (110) and an object (104), comprising the steps: a) capturing (128), by means of a first 3D camera (106), an object position vector (116) relative to the first 3D camera (106), b) capturing (130), by means of a second 3D camera (152), an eye position vector (114) relative to the second 3D camera (152), c) providing (132) a predetermined relation vector (159) which indicates the position of the first 3D camera (106) relative to the second 3D camera (152), and d) calculating (134) the position vector (124) from the eye (110) to the object (104) based on the object position vector (116), the eye position vector (114) and the relation vector (159).A computer-implemented method (136) for environmentally synchronous display of image data on a display (100), comprising the steps of: a) determining (126) the position vector (124) between an eye (110) and an object (104) according to claim 1, b) detecting (138) the image data by the first 3D camera (106) according to claim 1, c) determining (140) intersection points between the position vector (124) and a projection surface (102) of the display (100), and if at least one intersection point exists, d) generating (144) partial image data from the image data based at least in part on the position vector (124) such that the partial image data is perceived environmentally synchronously from the perspective of the eye (110) when displayed on the display (100), and e) displaying (146) the sub-image data on the display (100).Computer-implemented method according to Claim 1 or 2, characterized in that a set of objects is detected by means of object detection and the object (104) is selected from the set of objects on the basis of a safety-relevant classification of the objects.A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the computer-implemented method of any one of claims 1 to 3.A computer readable storage medium (168) comprising instructions which, when executed by a computer, cause the computer to perform the computer implemented method of any one of claims 1 to 3.A data carrier signal carrying the computer program product of claim 4.A data processing device (164) for the environment-synchronous representation of image data on a display (100) which is configured such that it can carry out the steps of a computer-implemented method according to one of claims 1 to 3, comprising: a) an evaluation unit (170), b) a display (100) which is communicatively connected to the evaluation unit (170), c) a first 3D camera (106) which is communicatively connected to the evaluation unit (170), and d) a second 3D camera (152) which is communicatively connected to the evaluation unit (170).The data processing device according to claim 7, characterized in that the display (100) is arranged on a structure of a vehicle (148) that blocks the view of a driver (150) of the vehicle (148) to an environment outside the vehicle (148).The device for data processing according to claim 8, characterized in that the first 3D camera (106) at least partially captures the environment of the vehicle (148) in the image data according to claim 2 or 3, which environment is not visible to the driver (150) through the structure of the vehicle (148).The data processing device according to claim 8 or 9, characterized in that the structure of the vehicle (148) is one of the A-pillar, the B-pillar, the C-pillar and the dashboard.The data processing device according to any one of claims 7 to 10, characterized in that the second 3D camera (152) is arranged in the interior of the vehicle (148) such that it can capture at least one eye (110) of the driver (150).The data processing device according to any one of claims 7 to 11, characterized in that the first 3D camera (106) and / or the second 3D camera (152) is configured with a laser, a radar, a lidar, at least two cameras or an ultrasonic sensor.Vehicle (148) comprising a data processing device (164) according to one of claims 7 to 12.

Images