Method and device for triggering a function of a vehicle using a captured image

Abstract

The invention relates to a method, in particular a computer-implemented method, for triggering a function of a vehicle using a captured image by means of an interior camera in the vehicle, the method including: (i) capturing a plurality of images of an object image represented on a physical object, in particular an electronic mobile device, by means of the interior camera; (ii) dividing each of the captured images of the plurality of images into a plurality of image sections; (iii) determining, for each image section of a first image of the plurality of images, whether the image sections each satisfy a specified criterion of an image parameter; (iv) determining a second image from the plurality of images, in which a second image section meets the specified criterion, if the determination for the image sections of the first image has shown that a first image section of the first image, on which substantially the same content as on the second image section is displayed, does not meet the specified criterion; (v) replacing the first image section of the first image with the second image section of the second image in the first image in order to obtain a corrected first image; (vi) triggering a function of a vehicle device of the vehicle using the corrected image.

Description

[0001] METHOD AND DEVICE FOR TRIGGERING A FUNCTION OF A

[0002] VEHICLE USING A CAPTURED IMAGE

[0003] The present invention relates to a method, in particular a computer-implemented method, a device, a motor vehicle and a computer program for triggering a function of a vehicle using a captured image.

[0004] Modern vehicles are increasingly equipped with advanced driver assistance systems (ADAS) designed to support the driver in specific driving situations. This support ranges from simply displaying potentially relevant information (e.g., issuing a warning via a lane departure warning system) to semi-autonomous interventions (e.g., regulating the torque applied to the wheel axles via an anti-lock braking system) and even fully autonomous interventions in the vehicle's control (e.g., adaptive cruise control via a distance control system, ACC).

[0005] The basis for such driver assistance systems is usually sensor data, such as signals provided by ultrasonic sensors, radar sensors or cameras, which are used to determine the current driving situation and to execute the function of the respective driver assistance system in response.

[0006] Furthermore, modern vehicles contain a multitude of additional functions and vehicle components that can be activated manually, semi-automatically, or automatically. These include windshield wipers, power windows, ventilation, heating, window heating, radio settings, navigation systems, and more. A navigation system can display a route as well as related traffic information such as speed limits, tunnels, and toll roads. These exemplary vehicle functions can typically be activated or controlled directly via a control element or dial, or via a display, particularly a central information display (CID). Additionally, a camera inside the vehicle can capture images of the interior or its occupants.For example, a gesture made by an occupant can be captured using image technology, and this gesture can in turn start or control a function, such as one of the vehicle functions described above.

[0007] Furthermore, user-defined information can be transferred to the vehicle via electronic mobile devices, such as vehicle keys or other electronic mobile devices. This user-defined information can include user-defined vehicle settings, such as seat position or interior temperature. This allows for personalized vehicle settings.

[0008] In today's world, it's often necessary to react quickly, easily, and intuitively to current situations. For example, a user might receive an image of a map section on their mobile device showing street names and a building where a meeting point has been arranged. To obtain route guidance to the building via an in-vehicle navigation system, the user would need to enter a street name or address into the system. A simpler way to transfer the image information to the navigation system would be desirable.

[0009] The present invention is based on the objective of providing a simple way to use information represented on an image of a physical object to trigger a function of a vehicle.

[0010] A solution to this problem is achieved according to the teaching of the independent claims. Various embodiments and further developments of the present invention are the subject of the dependent claims.

[0011] A first aspect of the solution concerns a method, particularly a computer-implemented method, for triggering a function of a vehicle using an image captured by an interior camera of the vehicle, comprising: (i) capturing a plurality of images of an object image displayed on a physical object, in particular an electronic mobile device, by the interior camera; (ii) dividing each of the captured images of the plurality of images into a plurality of image sections; (iii) determining, for each image section of a first image of the plurality of images, whether the image sections each satisfy a predetermined criterion of an image parameter;(iv) Determining a second image from the majority of images, in which a second image section fulfills the specified criterion, if the determination of the image sections of the first image has shown that a first image section of the first image, on which essentially the same content as on the second image section is depicted, does not fulfill the specified criterion; (v) Image-technical replacement of the first image section of the first image with the second image section of the second image in the first image in order to obtain a corrected first image; (vi) Triggering a function of a vehicle device of the vehicle using the corrected image.

[0012] Any terms used herein, such as "comprises," "includes," "features," "has," "with," or any other variant thereof, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or features a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent in such method or apparatus.

[0013] Furthermore, unless explicitly stated otherwise, "or" refers to an inclusive "or" and not an exclusive "or". For example, a condition A or B is satisfied by one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0014] The terms "ein" or "eine," as used here, are defined as "one or more." The terms "ein anderer" and "ein Weitere," as well as any other variant thereof, are to be understood as "at least one more."

[0015] The term "plural," as used here, is to be understood as "two or more." The terms "configured" or "set up" to perform a specific function (and their respective variations), as used here, mean that the corresponding device is already in a configuration or setting in which it can perform the function, or at least that it is adjustable—i.e., configurable—so that it can perform the function after appropriate adjustment. Configuration can be achieved, for example, by adjusting parameters of a process sequence or by using switches or similar devices to activate or deactivate functionalities or settings. In particular, the device can have several predefined configurations or operating modes, so that configuration can be carried out by selecting one of these configurations or operating modes.

[0016] The term "image sensor," as used here, refers specifically to an electro-optical sensor, particularly one in a camera, capable of capturing a scene. Specifically, the image sensor can be configured to detect or measure electromagnetic signals and convert them into electrical signals. These sensors can be charge-coupled sensors, also known as CCDs (Charge-Coupled Devices), radar sensors, lidar sensors, or other types of sensors. In particular, the image sensor can be a 3D image sensor capable of capturing a scene in three spatial dimensions, thus enabling, in particular, the measurement of distances within the scene.Furthermore, such a sensor can also be configured to emit electromagnetic signals to an object and to detect the electromagnetic signals reflected back from that object, so that information about the object can be obtained from the emitted and reflected electromagnetic signals in a subsequent analysis. Such sensors are used in so-called TOF (Time-of-Flight) cameras.

[0017] The term "control unit," as used here, refers in particular to an electronic device that controls the operation of a vehicle system, especially by means of a processor, particularly a CPU. Specifically, the control unit may include a transmitter and receiver for sending and receiving wireless signals, such as electromagnetic signals, and / or wired signals, such as electrical signals, via cables. Such a control unit may, in particular, include a microprocessor for analyzing received and / or previously stored data and / or for initiating a control process.

[0018] The term "image parameter," as used here, refers specifically to a parameter or criterion that describes the optical quality of an image. In particular, a parameter can be image sharpness, contrast, exposure, color saturation, or another characteristic.

[0019] The term "essentially the same," as used here, means in particular that two pieces of content, especially two images from two image sections, differ from each other by no more than 10%, and in particular by no more than 5%. Specifically, two essentially identical images or images mean that images of an object taken in rapid succession, i.e., typically a multiple of images taken within one second, may differ from each other due to a minimal, unintentional change in the orientation of the camera or the object.

[0020] The term "physical object," as used here, refers in particular to an object that is physically tangible and on or through which an image visible to a user can be displayed. Specifically, the physical object can be an electronic mobile device, such as a smartphone or tablet, whose displays can show the visible image. Likewise, the physical object can be an object with an optically passive surface, i.e., a non-luminescent surface, in particular a piece of paper, plastic, or a piece of another material on whose surface the image is printed.

[0021] The term "electronic mobile device," as used here, refers in particular to a smartphone, tablet computer, or mobile computer, each of which has one or more transmitting and receiving units for sending and receiving signals. These signals may be electromagnetic signals transmitted and received using a mobile communication standard, in particular Bluetooth, WLAN, or 2G, 3G, LTE, or 5G. It is also conceivable that the signals may be transmitted and received via a wired connection.

[0022] The term "function" or "functions," as used here, refers in particular to vehicle functions, such as infotainment system functions or other vehicle functions, or driving-related functions like route calculation and display by a navigation system. However, it also specifically includes non-driving-related functions or applications that can be executed in the vehicle but are not, or at least not directly, related to the vehicle or a journey, and instead serve purposes such as entertainment, like computer or video games or other applications.

[0023] The term "vehicle equipment," as used here, refers to a device that can execute predetermined instructions during the operation of a vehicle. In particular, this can include vehicle equipment used in a driver assistance system, an infotainment application, a drive system, or a chassis system. It can also include, in particular, a navigation system or a display located in the vehicle that allows vehicle functions to be controlled.

[0024] The term "position," in particular the position of an object, especially an electronic mobile device or an indoor camera, as used here, refers specifically to the position of a representative point of the respective object. This can be, in particular, the center point of a cross-sectional area or the object's center of gravity.

[0025] The method described in the first aspect enables a vehicle function to be triggered by an image captured by an interior camera and displayed on a physical object, particularly an electronic mobile device. Specifically, this allows for the enhancement of an image captured by the vehicle's interior camera with respect to a specific image parameter of a portion of the image. This enables better evaluation of the enhanced image's content and more reliable activation of the vehicle function triggered by content displayed in the image. This can be advantageous when the captured image is relatively small, such as an image on a smartphone display, and is subject to movement, as is the case with a smartphone held in a user's hand.

[0026] Preferred embodiments of the method are described below, which, unless expressly excluded or technically impossible, can be combined with each other and with the other aspects described.

[0027] In some embodiments, a distance between the physical object and the indoor camera is determined, particularly using a time-of-flight (TOF) camera, and the criterion is not met if the determined distance is less than a predetermined distance. In particular, the predetermined distance can be a minimum distance of a lens on the indoor camera, such that at a distance less than the minimum distance, the lens cannot produce a sharp image. This allows for a simple and reliable determination of whether one or more image sections can be captured sharply.

[0028] In some embodiments, the method further comprises: (i) determining the position of the interior camera; (ii) determining the position of the physical object using the determined position of the interior camera and the determined distance, wherein the criterion is not met if the position of the physical object lies outside a predetermined spatial range. In particular, the positions of the interior camera and the reference object can be determined by a known measurement and positioning of objects inside a vehicle, or subsequently read from a database. The lens of the interior camera defines an opening angle within which an image can be captured. In particular, the predetermined spatial range can represent this opening angle. The spatial range increases with increasing distance from the lens.This embodiment makes it possible to determine whether the physical object is located wholly or partially outside this spatial area, and if so, whether the image of the physical object can be captured. If not, at least the uncapturable portion of the image can be replaced.

[0029] In some embodiments, the first image section of the first image is replaced by a new image section, which is created by superimposing a plurality of image sections, each representing essentially the same image content, from the majority of the captured images. This can achieve a particularly good improvement in the image section being replaced. In particular, this can enable an improvement with regard to the exposure of the image.

[0030] In some embodiments, an angle is determined between an image plane of the object image and a sensor plane of an image sensor of the indoor camera. The angle is 0° when the image plane and the sensor plane are parallel to each other, and the criterion is not met if the determined angle is greater than a predefined value. This makes it possible to detect when the physical object and the object image displayed on it are tilted too much with respect to the sensor plane, and therefore a captured image does not meet the criterion.

[0031] In some embodiments, the criterion is not met if the image sharpness, as an image parameter, is lower than a predefined sharpness level. This allows a first image section to be replaced by a second image section with higher sharpness.

[0032] In some embodiments, the criterion is not met if the average image contrast, as an image parameter, is lower than a predefined image contrast. This allows a first image section to be replaced by a second image section with a higher average image contrast.

[0033] In some embodiments, the corrected first image is checked to determine whether a intended function of a vehicle device can be triggered using the corrected first image, and whether the function is triggered after receiving a confirmation signal. This allows it to be determined in advance whether the function can be triggered, and subsequently, in particular by the user, an input can be made that triggers the confirmation signal.

[0034] In some versions, a message is displayed on the vehicle's screen after the check if the function cannot be triggered. This allows the user to hold the physical object in a different orientation or at a different distance within the field of view of the interior camera again in order to trigger the function.

[0035] A second aspect of the solution concerns a device for triggering a function of a vehicle using an image captured by an interior camera of the vehicle, wherein the device is configured to perform the method according to the first aspect.

[0036] In some embodiments, (i) the indoor camera is configured to capture a plurality of images of an object image displayed on a physical object, in particular an electronic mobile device; (ii) wherein the device further comprises a control unit configured to perform the following steps: (ii-1) divide each of the captured images of the plurality of images into a plurality of image sections; (ii-2) determine, for each image section of a first image of the plurality of images, whether the image sections each satisfy a predetermined criterion of an image characteristic;(ii-3) Determining a second image from the plurality of images, in which a second image section satisfies the specified criterion, if the determination of the image sections of the first image has shown that a first image section of the first image, on which substantially the same content as on the second image section is depicted, does not satisfy the specified criterion; (ii-4) Image-technical replacement of the first image section of the first image with the second image section of the second image in the first image in order to obtain a corrected first image; (ii-5) Triggering a function of a vehicle device of the vehicle using the corrected image.

[0037] A third aspect of the solution concerns a vehicle, in particular a motor vehicle, comprising a device according to the second aspect. A fourth aspect of the solution concerns a computer program with instructions which, when executed on a device according to the second aspect, cause it to execute the method according to the first aspect.

[0038] The computer program can be stored, in particular, on a non-volatile data carrier. Preferably, this is a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program itself is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can exist as a file on a data processing unit, in particular on a server, and be downloadable via a data connection, for example, the Internet or a dedicated data connection, such as a proprietary or local network. Furthermore, the computer program can comprise a plurality of interacting individual program modules. The modules can, in particular, be configured, or at least be usable, in such a way that they function in the sense of distributed computing (i.e., distributed computing)."Distributed computing" is performed on different devices (computers or processor units) that are geographically separated from each other and connected via a data network.

[0039] The features and advantages explained in relation to the first aspect of the solution also apply to the other aspects described.

[0040] Further advantages, features and application possibilities will result from the following description of preferred embodiments in conjunction with the figures.

[0041] This shows

[0042] Fig. 1 schematically shows a top view of a motor vehicle;

[0043] Fig. 2 schematically shows a first image and a second image; and

[0044] Fig. 3 schematically illustrates a flowchart to demonstrate one embodiment of a method. The same reference symbols are used throughout the figures for the same or corresponding elements.

[0045] Figure 1 schematically shows a motor vehicle 100 according to one embodiment. The motor vehicle 100 has an interior camera 110 which is connected to a control unit 120 via a signal connection, thereby enabling the control of the interior camera 110 and the evaluation of images captured by the interior camera 110. Furthermore, the control unit 120 is connected to vehicle equipment (not shown here) of the motor vehicle 100 via a signal connection in order to trigger individual functions of this vehicle equipment.

[0046] Furthermore, a schematic representation shows an electronic mobile device 140, in particular a smartphone, with a display (not shown here). The display of the electronic mobile device 140 is oriented towards the indoor camera 110. This is indicated by the double arrow between the electronic mobile device 140 and the indoor camera. The indoor camera 110 can capture images from the smartphone 140 or content displayed on its screen. This is further illustrated by the direction of the double arrow pointing towards the indoor camera. The indoor camera can also be configured as a time-of-flight (TOF) camera. The TOF camera can transmit signals to the electronic mobile device 140, as illustrated by the arrow pointing towards the electronic mobile device 140. Using a reflected signal of the transmitted signal, a distance or...A distance between the interior camera 110 and the electronic mobile device 140 is determined. This determination can be carried out by the control unit 120.

[0047] Furthermore, a vehicle display 130, in particular a so-called "central information display" (CID), is arranged on the dashboard of the vehicle 200. The vehicle display 130 is connected to the control unit 120 via a signal connection. This allows the control unit 120 to control the output of messages, especially those related to the results of an evaluation of the captured images, by the vehicle display 130. Figure 2 schematically depicts a first image 210 (in relation to the plane of the drawing above) and a second image 220 (in relation to the plane of the drawing below), wherein the first image 210 and the second image 220 were each captured by an interior camera 110 of a motor vehicle 100 according to Figure 1. The first image 210 and the second image 220 were captured in close succession, and the first image 210 and the second image 220 show essentially the same content.The first image 210 is composed of the image sections or segments SA1 to SA6 and the second image is composed of the image sections or segments SB1 to SB6.

[0048] The arrow pointing from image section SB4 of the second image 220 to image section SA4 of the first image 210 schematically illustrates that image section SA4 is to be replaced by image section SB4. This can be done, for example, if image section SA4 is technically blurry, but image section SB4 is sharp. Replacing the two sections then improves the overall image quality of the first image 210.

[0049] This is described in more detail in the procedure shown in Fig. 3.

[0050] Fig. 3 schematically shows a flowchart 300 to illustrate an embodiment of a method for triggering a function of a motor vehicle 100 using an image captured by an interior camera 110 of the motor vehicle 100.

[0051] In a first step S310 of the procedure, a plurality of images 210, 220 of an object image displayed on an electronic mobile device 140 are captured by the indoor camera 110. Since the object image is captured in each instance, the content displayed in the respective images 210, 220 of the plurality of images is essentially the same or identical. In particular, the images are captured sequentially at short intervals. Typically, known cameras can capture a series of multiple images, for example, 5 to 10 images, within one second.

[0052] In a further step S320 of the procedure, each of the captured images 210, 220 of the majority of the images is divided into a plurality of image sections SA1-SA6, SB1-SB6. This can be done by an image processing algorithm in the control unit.

[0053] In a further step S330 of the procedure, it is determined for each image section SA1-SA6 of a first image 210 and the majority of images 210, 220 whether the image sections each meet a predefined criterion of an image parameter. This can be done using appropriate algorithms, particularly with regard to image sharpness, color saturation, and exposure. With known algorithms, for example, it is possible to automatically color-code overexposed areas.

[0054] In a further step S340 of the procedure, a second image 220 is determined from the majority of images 210, 220, in which a second image section SB1-SB6 fulfills the specified criterion, if the determination of the image sections of the first image 210 has shown that a first image section SA1-SA6 of the first image 210, on which essentially the same content as on the second image section SB1-SB6 is shown, does not fulfill the specified criterion.

[0055] In a further step S350 of the procedure, the first image section SA1-SA6 of the first image 210 is replaced by the second image section SB1-SB6 of the second image 220 in the first image in order to obtain a corrected first image.

[0056] In a further step S360 of the procedure, a function of a vehicle device in motor vehicle 100 is triggered using the corrected image. For example, the function could be route calculation if named streets and buildings are shown in the image.

[0057] While at least one exemplary embodiment has been described above, it should be noted that a large number of variations exist. It should also be noted that the described exemplary embodiments are merely non-limiting examples, and it is not intended to restrict the scope, applicability, or configuration of the devices and methods described herein. Rather, the preceding description will provide the person skilled in the art with guidance for implementing at least one exemplary embodiment. It is understood that various modifications to the function and arrangement of the elements described in an exemplary embodiment can be made without deviating from the subject matter defined in the appended claims and their legal equivalents.

[0058] REFERENCE MARK LIST

[0059] 100 motor vehicles

[0060] 110 Interior camera

[0061] 120 Control unit

[0062] 130 vehicle display

[0063] 140 Electronic mobile device

[0064] 210, 220 First, second image

[0065] SA1-SA6 Image excerpts of the first image

[0066] SB1-SB6 Image excerpts of the second image

[0067] 300 Flowchart

[0068] S310 Capture multiple images

[0069] S320 Splitting into image sections

[0070] S330 Determine whether criterion is met

[0071] S340 Determining the second image section of the second image

[0072] S350 Image Technology Replacement

[0073] S360 Triggering a function

Claims

REQUIREMENTS 1. Method for triggering a function of a vehicle (100) using an image captured by an interior camera (110) of the vehicle (100), comprising: Capturing a plurality of images (210, 220) of an object image displayed on a physical object (140) by the indoor camera (110); Dividing each of the captured images (210, 220) of the majority of images into a plurality of image sections (SA1-SA6, SB1-SB6); Determine for each image section (SA1-SA6) of a first image (210) of the majority of images (210, 220) whether the image sections each meet a predefined criterion of an image parameter; Determining a second image (220) from the majority of images (210, 220) in which a second image section (SB1-SB6) fulfills the specified criterion, if the determination of the image sections of the first image (210) has shown that a first image section (SA1-SA6) of the first image (210), on which essentially the same content as on the second image section (SB1-SB6) is depicted, does not fulfill the specified criterion; Image-technical replacement of the first image section (SA1-SA6) of the first image (210) by the second image section (SB1-SB6) of the second image (220) in the first image in order to obtain a corrected first image; Triggering a function of a vehicle equipment of the vehicle (100) using the corrected image.

2. Method according to claim 1, wherein a distance between the physical object (140) and the indoor camera (110) is determined, and wherein the criterion is not met if the determined distance is less than a predetermined distance.

3. The method according to claim 2, further comprising: Determining the position of the interior camera (110); Determining the position of the physical object (140) using the determined position of the indoor camera (110) and the determined distance, and wherein the criterion is not met if the position of the physical object (140) is outside a specified spatial area.

4. Method according to one of the preceding claims, wherein the first image section of the first image is replaced by a new image section which is replaced by a superimposition of a plurality of image sections, each representing essentially the same image content, from the plurality of the captured images.

5. Method according to one of the preceding claims, wherein an angle between an image plane of the object image and a sensor plane of an image sensor of the indoor camera (110) is determined, wherein the angle is 0° when the image plane and the sensor plane are arranged parallel to each other, and wherein the criterion is not met if the determined angle is greater than a predetermined value.

6. Method according to one of the preceding claims, wherein the criterion is not met if an image sharpness as an image parameter is less than a predetermined image sharpness.

7. Method according to one of the preceding claims, wherein the criterion is not met if an average image contrast as an image parameter is less than a predetermined image contrast.

8. Method according to one of the preceding claims, wherein the corrected first image is checked to determine whether a provided function of a vehicle device can be triggered using the corrected first image, and the function is triggered after receiving a confirmation signal.

9. Method according to claim 8, wherein, following the check, a message is displayed on a vehicle display (130) of the vehicle if the function cannot be triggered.

10. Device for triggering a function of a vehicle (100) using an image captured by an interior camera (110) of the vehicle (100), comprising the device being configured to carry out the method according to one of the preceding claims.

11. Device according to claim 10, wherein the interior camera (110) is configured to capture a plurality of images (210, 220) of an object image displayed on a physical object (140); wherein the device further comprises a control device (120) configured to perform the following steps: Dividing each of the captured images (210, 220) of the majority of images into a plurality of image sections (SA1-SA6, SB1-SB6); Determine for each image section (SA1-SA6) of a first image (210) of the majority of images (210, 220) whether the image sections each meet a predefined criterion of an image parameter; Determining a second image (220) from the majority of images (210, 220) in which a second image section (SB1-SB6) fulfills the specified criterion, if the determination of the image sections of the first image (210) has shown that a first image section (SA1-SA6) of the first image (210), on which essentially the same content as on the second image section (SB1-SB6) is depicted, does not fulfill the specified criterion; Image-technical replacement of the first image section (SA1-SA6) of the first image (210) by the second image section (SB1-SB6) of the second image (220) in the first image in order to obtain a corrected first image; Triggering a function of a vehicle equipment of the vehicle (100) using the corrected image.

12. Vehicle (100) comprising a device according to claim 10 or 11.

13. Computer program with instructions which, when executed on a device according to one of claims 10 to 11, cause the device to execute the method according to one of claims 1 to 9.

Images