Method and device for processing image data associated with the passenger compartment of a vehicle

The method employs semantic segmentation and neural networks to accurately determine camera position and orientation within vehicle compartments, addressing variability in vehicle types and lighting, thereby improving calibration efficiency and robustness.

FR3149413A1Pending Publication Date: 2024-12-06ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
FR2024005401
Authority / Receiving Office
FR · FR
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-31
Filing Date
2024-05-27
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing methods struggle to accurately determine the position and orientation of cameras within vehicle passenger compartments, particularly due to variations in vehicle type, camera installation, and lighting conditions, which complicates calibration and correct orientation.

Method used

A computer-implemented method using semantic segmentation, neural networks, and image classification to identify regions within vehicle passenger compartments, allowing for the determination of camera position and orientation by analyzing image data and transforming it into vectors for robust decision-making.

Benefits of technology

Enables accurate determination of camera position and orientation, even in varying conditions, enhancing calibration efficiency and robustness against changes in camera type and lighting, while maintaining stability across different vehicle types and lighting conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Title: Method and Device for Processing Image Data Associated with the Interior of a Vehicle. A computer-implemented method for processing image data (BD-IR) associated with the interior (10-IR) of a vehicle (10) by a camera (1) consisting of: - providing (100) a semantic segmentation (SS-BD) of the image data (BD-IR), - determining (102) at least one first region associated with the image data (BD-IR), for example, a foreground region (REG-1) based on the semantic segmentation (SS-BD), - determining (104) at least one of the following based on the first region (REG-1): a) the position (1-POS) of the camera (1) in the interior (10-IR), b) orienting (1-OR) the camera (1) in the interior (10-IR). Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Title of the invention: Method and device for processing image data associated with the passenger compartment of a vehicle. FIELD OF THE INVENTION

[0001] The present invention relates to a method and device for processing image data associated with the passenger compartment of a vehicle.

[0002] DESCRIPTION AND ADVANTAGES OF THE INVENTION

[0003] The invention relates to a computer-implemented method for processing image data associated with the passenger compartment of a vehicle, for example, a motor vehicle by a camera, consisting of providing a semantic segmentation of the image data, determining at least a first region associated with the image data, for example, a foreground region based on the semantic segmentation, determining at least one of the following based on at least the first region: a) position of the camera in the passenger compartment of the vehicle, b) orientation of the camera in the passenger compartment of the vehicle.

[0004] According to another characteristic, the provision of semantic segmentation has at least one of the following steps:

[0005] a) determine the semantic segmentation, for example, using a neural network,

[0006] b) receive the semantic segmentation, for example, from another unit,

[0007] c) call the semantic segmentation, for example, from another unit.

[0008] According to another characteristic, the process consists of:

[0009] - use the camera position or orientation for at least one of the steps following:

[0010] a) exploit the image data,

[0011] b) check the installation location or installation position of the camera in the passenger compartment,

[0012] c) calibrate the camera or a system or method using the camera image data.

[0013] According to one characteristic, the process consists of:

[0014] - determine a first region associated with the image data and at least one another region associated with image data from semantic segmentation,

[0015] - determine the camera position or camera orientation in the passenger compartment of the vehicle based on the first region associated with the image data and at least one other region associated with the image data.

[0016] According to another characteristic, the first region or at least one other region characterizes one of the following elements:

[0017] a) foreground region,

[0018] b) background region and, for example, the foreground region includes at least one of the following elements:

[0019] a) passenger compartment ceiling,

[0020] b) seating area, with, for example, the background region not having a ceiling, or cabin ceilings, or one or more seating areas.

[0021] According to another feature, the process consists of:

[0022] perform a classification of image data into several classes, for example, by image points, the classes being characterized by at least one of the following elements of the vehicle's interior:

[0023] a) front seat backrest,

[0024] b) rear seat backrest,

[0025] c) front seat surface,

[0026] d) rear seat surface,

[0027] e) window,

[0028] f) ceiling,

[0029] g) center console,

[0030] h) persons,

[0031] i) other objects and, optionally,

[0032] - use classification, for example, to determine at least a first region associated with image data.

[0033] According to one characteristic, the process consists of:

[0034] - associate the image points of a first group of classes to the first region and in option,

[0035] - associating image points from a second group of classes with one or more second regions, the classes of the second group being different from the classes of the first group and optional,

[0036] - consolidate the first region and / or the second region.

[0037] According to another characteristic, the process consists of:

[0038] - transform into a vector, for example, a column vector, the image data characterizing the first and second regions, and whose elements respectively characterize a dominant region of a corresponding line of the first and second regions, with, for example, the transformation of the execution of a majority decision on the image data of a respective line of image data characterizing the first and second regions and, optionally,

[0039] - use the column vector for at least one of the following steps:

[0040] a) determine the position of the camera in the passenger compartment,

[0041] b) determine the orientation, for example, the vertical orientation of the camera in the passenger compartment.

[0042] According to another feature, the process comprises at least one of the following steps:

[0043] - provide one or more neural networks,

[0044] - train the neural network,

[0045] - determine the semantic segmentation of image data with the network neuronal.

[0046] According to other features, the invention relates to a method, for example, a computer-implemented method for training a neural network, for example, an artificial neural network, for example, a deep neural network, for the semantic segmentation of image data from a camera, comprising the steps of:

[0047] - provide training data that have image data associated with the less a vehicle interior, for example, that of a motor vehicle,

[0048] - train the neural network for semantic segmentation of image data associated with the passenger compartment of a vehicle, for example, a motor vehicle, and as an option,

[0049] - use the neural network to determine the semantic segmentation of the data camera image.

[0050] Depending on other characteristics, the training includes at least one of the following elements:

[0051] a) take into account different vehicles or different types of vehicles,

[0052] b) take into account different installation positions and / or orientations of the camera,

[0053] c) take into account different types of cameras and / or different lighting conditions.

[0054] According to other features, the invention relates to a computer-readable memory medium comprising instructions, the execution of which by the computer commands it to execute the process according to the embodiments.

[0055] The invention also relates to a computer program comprising instructions which, when the program is executed by the computer, command it to apply the process.

[0056] The invention also relates to a data support signal that is transmitted and / or characterizes the computer program.

[0057] The invention also relates to the application of the method according to the embodiments or of the device according to the embodiments or of the computer-readable memory medium according to the embodiments or of the computer program depending on the embodiment or the data carrier signal, depending on the embodiment for at least one of the following steps:

[0058] a) determine the position of a camera in the vehicle's passenger compartment,

[0059] b) determine the orientation of one or more cameras in the vehicle's passenger compartment,

[0060] c) exploit the camera's image data,

[0061] d) check the installation location or installation position of the camera in the vehicle's passenger compartment,

[0062] e) calibrate the camera or a system or method using the camera's image data,

[0063] f) increase robustness against a change in camera type,

[0064] g) to abstract from a scene in the passenger compartment of the vehicle. Brief description of the drawings

[0065] The present invention will be described in more detail below with the aid of examples of embodiments of the computer-implemented method for processing image data associated with a vehicle interior, as shown in the accompanying drawings in which,

[0066] [Fig. 1] Simplified flowchart of an embodiment of the invention,

[0067] [Fig.2] Simplified block diagram of an embodiment,

[0068] [Fig.3] simplified flowchart diagram of an embodiment,

[0069] [Fig.4A] schematic aspect of a semantic segmentation of a form of realization,

[0070] [Fig.4B] diagram of a semantic segmentation of a realization form,

[0071] [Fig.5A] Schematic of regions according to embodiment examples,

[0072] [Fig.5B] Schematic of regions according to embodiment examples,

[0073] [Fig.6A] Schematic of regions according to embodiment examples,

[0074] [Fig.6B] Schematic of regions according to embodiment examples,

[0075] [Fig.6C] Schematic of regions according to embodiment examples,

[0076] [Fig.7] Simplified flowchart of an embodiment,

[0077] [Fig.8A] Simplified flowchart of an embodiment,

[0078] [Fig.8B] Simplified flowchart of an embodiment,

[0079] [Fig.9] Simplified block diagram of an embodiment,

[0080] [Fig. 10] Simplified flowchart of an embodiment,

[0081] [Fig. 11] Simplified block diagram of an embodiment,

[0082] [Fig. 12] diagram of the application of an embodiment form.

[0083] DESCRIPTION OF EMBODIMENT METHODS OF THE INVENTION

[0084] Illustrative embodiments (Figures 1, 2) relate to a method, for example, a computer-implemented method for processing data BD-IR image from a camera 1, associated with the passenger compartment 10-IR of a vehicle, for example, a motor vehicle 10, and consisting of:

[0085] - provide 100 ([Fig.1]) a semantic SS-BD segmentation of image data BD-IR,

[0086] - determine at least one first REG-1 region associated with the image data BD-IR, for example, the foreground region based on SS-BD semantic segmentation,

[0087] - determine at least one of the following elements based on at least the first region REG-1:

[0088] a) 1-POS position of camera 1 in the vehicle's passenger compartment 10-IR,

[0089] b) 1-OR orientation of camera 1 in the vehicle's interior 10-IR. This allows, for these embodiments, verification of the correct positioning or orientation of camera 1 in the vehicle's interior 10-IR.

[0090] In embodiment examples ([Fig. 1]), providing (100) the SS-BD semantic segmentation of at least one of the elements consists of:

[0091] a) determine 100a the SS-BD semantic segmentation, for example, using a neural network,

[0092] b) receive 100b the SS-BD semantic segmentation, for example, from another (unrepresented) unit,

[0093] c) call 100c the SS-BD semantic segmentation, for example, of another unit.

[0094] In embodiments given by way of example ([Fig. 1]), the process consists of:

[0095] - use 106 the 1-POS position or the 1-OR orientation of camera 1 for at least one of the following steps:

[0096] a) exploit AW BD-IR image data,

[0097] b) check UEB the installation location or installation position of camera 1 in the vehicle's passenger compartment 10-IR,

[0098] c) calibrate KAL camera 1 or one of the systems or methods using the BD-IR image data from camera 1.

[0099] In the embodiment examples ([Fig.3]) the process consists of:

[0100] - determine 110 the first REG-1 region associated with the BD-IR image data (for example, by at least part of the characterized image data) and at least one other REG-2 region associated with other BD-IR image data (for example, by at least one other part of the characterized image data) based on SS-BD semantic segmentation ([Fig.1]),

[0101] - determine 112 ([Fig.3]) the 1-POS position of camera 1 or the 1-OR orientation of camera 1 in the vehicle's 10-IR interior based on the first region REG-1 associated with BD-IR image data and on at least one other REG-2 region associated with BD-IR image data.

[0102] In embodiments, by way of example, the first REG-1 region or at least one other REG-2 region characterizes one of the following:

[0103] a) foreground region,

[0104] b) background region, for example, the foreground region which includes at least one of the following:

[0105] a) ceiling of the vehicle passenger compartment,

[0106] b) seat or seating area with, for example, without the floor area, nor one or more ceilings of the vehicle passenger compartment, nor one or more seat or seating areas.

[0107] For example, the first region REG-1 characterizes a foreground region with, for example, the roof of the vehicle's passenger compartment and a seating area. Further details are described below, for example, with reference to Figures 4A, 4B, 5A, 5B.

[0108] Figure 4A shows schematic aspects of semantic segmentation according to implementation examples, for example, according to block 100 of Figure 1, based on the BD-IR image data of camera 1. For example, the BD-IR image data of one or more camera images or the 10-IR cabin images derived from them (for example, by image processing) (Figure 2). In the semantic segmentation given as an example, as shown in Figure 4A, each image point is associated with one or more possible classes, thus providing a classification by image points of a camera image. In other words, the semantic segmentation can also characterize, as an image, the values ​​associated with the different image points characterizing the respective class based on the classification mentioned above. For example, the image points in area B1 according to Figure 1.4A] are associated with a first part (e.g. the ceiling) of the 10-IR cabin; the image points in the B2 zone(s) are associated with a second part (another part, e.g. the seats) of the 10-IR cabin, etc.

[0109] [Fig.4B] shows the schematic aspects of a semantic segmentation analogous to that of [Fig.4A] based, however, on a different orientation, for example, vertical, of the camera 1. [Fig.4B] shows, certainly, areas of images BD associated with the seat, but not the area of ​​images B1 associated with the ceiling.

[0110] In the embodiment examples ([Fig.7]) the process consists of:

[0111] - perform 120 a KLASS point-image classification of image data BD-IR in several classes, for example, the classes being characterized by at least one of the following elements of the 10-IR passenger compartment: a) front seat backrest, b) rear seat backrest, c) front seating surface, d) rear seating surface, e) window f) ceiling (see, for example, also area B1 of [Fig. 4A]), g) center console, h) persons, i) other objects and optionally,

[0112] - use 122 the KLASS classification, for example to determine at least 102 one of the REG-1, REG-2 regions associated with BD-IR image data.

[0113] In the embodiment examples of [Fig.8A], the process consists of:

[0114] - associating 130 of the image points of a first group GR-KL-1 of classes with the first region REG-1 and optionally:

[0115] - associate 132 of the image points of a second group GR-KL-2 of classes to a or second regions REG-2; the classes of the second group are, for example, different from the classes of the first group and optionally,

[0116] - consolidate 134 (for example, combine partial areas) the first REG-1 region or the second region REG-2.

[0117] Figure 5A shows, for example, the first region REG-1, which does not necessarily include the attached zones Bl and B2 that characterize, for example, different foreground zones VG-1 and VG-2. The first region REG-1 according to Figure 5A is based, for example, on association 130 according to Figure 8A. For example, the classes of image points that have been characterized by references Bl and B2 in Figure 5A can be associated with the first region REG-1 in certain examples; that is to say, for example, the image points that belong to the seating surfaces or seats, the image points that belong to the steering wheel and the dashboard cover, the image points that belong to the ceiling, etc.For example, the second region REG-2 can be associated, for certain realization examples, with an HG background which, for other realization forms, characterizes all other classes of image points, that is to say, the object points which have not been associated with the first region REG-1.

[0118] [Fig.5B] shows, for example, consolidated areas Bl', B2' or regions REG-1', REG-2' obtained by consolidation 134 according to [Fig.8A] and which, for example, still allow us to distinguish between the foreground VG-1, VG-2 (region REG-1') and the background HG', REG-2', which is sufficient for other examples of realization with certain operations or applications but which have nevertheless reduced the complexity with regard to the representation of [Fig.5A].

[0119] In the case of the embodiment examples in [Fig. 8B], the process consists of:

[0120] - transforming 135 the characteristic image data of the first REG-1 region and the second region REG-2 into a vector, for example, a column vector SV whose elements respectively characterize a dominant region of a corresponding row that characterizes the image data characteristic of the first region REG-1 and the second region REG-2; for example, transforming 135 consists of executing 135a one of the majority decisions concerning the image data of a respective row of the image data characterizing the first region REG-1 and the second region REG-2 and optionally

[0121] - use the SV column vector 137 for at least one of the following steps:

[0122] a) determine the 1-POS position of camera 1 in the vehicle's 10-IR interior,

[0123] b) determine the 1-OR orientation, for example, the vertical orientation of camera 1 in the vehicle's 10-IR passenger compartment.

[0124] Figure A shows for this purpose an example of an image analogous to that of Figure 5B with consolidated areas REG-1', REG-2' associated with a foreground VG-1, VG-2 or a background HG. To transform it according to Figure 8B, one can, for example, for each line of the image characterized by Figure A, have an element of the column vector SV mentioned above, for example, by a majority decision. For this, in a relevant line Z of the image characterized according to Figure A, one determines to which region REG-1, REG-2 or to which area VG-1, VG-2, HG' belong the image points of the line Z considered, and this provides the value of the element E in the column vector SV.

[0125] In the embodiment examples, the column vector can be used, for example, by determining, from neighboring vector elements of different values, the corresponding rows of which are associated with different regions (see the double arrows a1, a2 or their position in the column vector SV). On this basis, the vertical orientation of the camera 1 ([Fig. 2]) in the cabin 10-IR can be determined for other embodiment examples.

[0126] Figure 6B shows an example of an image analogous to that of Figure 6A; however, the image in Figure 6B nevertheless shows a foreground region VG1 associated with the ceiling of the 10-IR cabin in a relatively large part of the image. This is correspondingly reflected by a column vector not referenced to Figure 6B and having a relatively large number of elements corresponding to the foreground region VG1. Based on the situation in Figure 6B or the corresponding column vector, one can, for example, conclude here that the vertical orientation of camera 1 (Figure 2) is incorrect because, for example, too large a proportion of image points are associated with the ceiling.

[0127] Figure 6C shows an example of an image analogous to that of Figure 6A but for an embodiment according to Figure 6B which gives a relatively balanced representation of the different regions VG-1, VG-2, HG or REG-1', REG-2'. This results in a column vector not referenced in Figure 6C.

[0128] In another embodiment, the SS-BD semantic segmentation ([Fig. 1]) can, for example, be used to identify the foreground regions VG1, VG2 ([Fig. 5A], 5B) which include, for example, the ceiling on one side and, for example, the seating area on the other side. The background region HG is defined as the region that belongs neither to the ceiling nor to the seating area.

[0129] As described previously, with reference to [Fig. 6A], for other embodiments, for the distribution into different image regions REG-1, REG-2, (for example [Fig. 6A]), proceeding from left to right, a respective class type can be determined with a maximum number of image points that do not belong to the ceiling, seat, or background regions. Then, for other embodiments given by way of example, a boundary can be defined (for example, between the cover region VG-1 and the seat region VG-2, for example, the double arrows a1, a2 according to [Fig. 6A]). In some embodiments, the boundary is located, at least substantially, in the middle of the two regions and is bordered by the background region HG'.For other realization examples, if the background region HG' (viewed, for example, in relative terms) is too large, this may indicate that either the seat region or the ceiling region has been covered. In this case, for other realization examples, the boundary can be adapted, for example, to the foreground region more heavily replaced in the images and preceded, for example, by a predefined offset (d) (see [Fig.6C]).

[0130] In the embodiment examples of [Fig.9], the process includes at least one of the following steps:

[0131] - provide 140 one or more NN neural networks,

[0132] - train 142 the NN neural network,

[0133] - determine 144 the SS-BD semantic segmentation of image data with the NN neural network.

[0134] Other embodiments according to [Fig. 10] relate to a method, for example, a computer-implemented method for training the neural network NN, for example, a deep artificial neural network for the semantic segmentation of BD-IR image data from a camera 1, comprising the steps of:

[0135] - provide 180 TD training data that include image data BD-IR associated with a 10-IR passenger compartment of at least one vehicle 10, for example, a motor vehicle,

[0136] - train 182 the NN neural network for semantic data segmentation of images associated with a vehicle interior, for example, of a motor vehicle, and, optionally,

[0137] - use 184 the NN neural network (e.g., trained) to determine the SS-BD semantic segmentation of camera image data.

[0138] In the embodiment examples of [Fig. 10], drive 180 includes at least one of the following elements:

[0139] a) take into account 182a the different vehicles and / or types of vehicles,

[0140] b) take into account 182b the different installation positions and / or orientations from camera 1,

[0141] c) take into account 182c the different types of camera and / or lighting conditions.

[0142] Other examples of embodiment according to [Fig.1 1] concern a device 200 for the application of the process to the different forms of embodiment.

[0143] For other embodiments according to [Fig. 1 1], the device 200 comprises:

[0144] - a computing installation (computer) 200 with at least one computing core, a memory installation 204 associated with the computing installation 202 to store at least temporarily one of the following:

[0145] a) DAT data (for example BD-IR image data or data derived from it, for example SS-BD semantic segmentation or data characterizing SS-BD semantic segmentation, classes obtained, regions obtained, etc.

[0146] b) a computer program PRG, for example, for carrying out the process according to the embodiments.

[0147] According to other embodiments, the memory installation 204 comprises a fugitive memory (for example, a working memory (RAM) 204a or a non-volatile memory (NVm-), (for example, a Flash-EEPROM) 204b or a combination of these with other types of memory not specified.

[0148] Examples of embodiments relate to a computer-readable SM memory support comprising PRG' instructions which, executed by a computer 202, command it to apply the process according to the embodiment forms.

[0149] Other examples of embodiment relate to the instructions of the computer program PRG, which, by the execution of the PRG program by a computer 202, command it to execute the process according to the embodiment forms.

[0150] Other embodiment examples relate to a DCS data carrier signal that the PRG program characterizes according to embodiment forms or transmits. The DCS data carrier signal is, for example, received by an optional data interface 206 of the device 200.

[0151] Other embodiment examples according to [Fig.12] relate to the use 300 of the method according to embodiments or of the device 200, according to embodiments or of the computer-readable memory medium SM according to embodiments or of the computer program PRG, PRG' according to embodiments or of the data carrier signal DCS according to embodiments for at least one of the following:

[0152] a) determine 301 the position of a camera 1 in the passenger compartment 10-IR of a vehicle ([Fig.2]),

[0153] b) determine 302 the orientation of one or more cameras 1 in the vehicle's interior 10-IR,

[0154] c) exploit 303 the BD-IR image data from camera 1,

[0155] d) check 304 the installation location or installation position of the camera in the vehicle's passenger compartment 10-IR,

[0156] e) calibrate 305 the camera or a system (not shown) or a method using the camera's image data,

[0157] f) increase 306 robustness with respect to camera type change,

[0158] g) abstract 307 a scene in the vehicle's 10-IR interior.

[0159] Other aspects and embodiments will be described above which can be taken in isolation or in combination with at least one other of the preceding aspects.

[0160] Other embodiments are used for verifying the camera's position in any vehicle interior. For example, semantic segmentation 100 ([Fig. 1]) can use a deep convolutional neural network, for example, of the type described in the following publication: "ImageNet Classification with Deep Convolutional Neural Networks".

[0161] Examples of implementation can be deduced, for example, from the emission of the neural network NN of the correct orientation of the camera 1 for any types of vehicle.

[0162] Examples of embodiments may, for example, help to solve the problem whereby for a variable installation position of a camera 1 in any vehicle 10 ([Fig.2]) or for the passenger compartment, the calibration of the camera 1 may be relatively difficult and thus not respond to conventional approaches such as, for example, the correct orientation of the camera which cannot be guaranteed.

[0163] In other embodiments, the vehicle passenger compartment 10-IR differs significantly between different vehicles and vehicle types, resulting, for example, in an unknown installation position that allows for more variations. The principle, according to the embodiments, can provide for the correct installation position despite possible variations.

[0164] According to other embodiments, the principle is relatively robust, for example, with regard to the replacement of camera types, which has, for example, the advantage in the event of a change in the optical elements of the camera, of not having to adapt the process to exploit the camera images.

[0165] According to other embodiments, the method operates, for example, during movement, for partial or complete occupancy of the seats. This facilitates, for example, calibration and also allows the function to be used during movement.

[0166] According to other embodiments, the abstraction of the scene from the camera image by SS-BD semantic segmentation ([Fig.1]) makes the process robust with respect to variations in lighting and can be used both day and night, with a NIR camera (camera operating in the near infrared).

[0167] According to other examples, the scene is removed from the cabin 10-IR insofar as, by semantic segmentation and association in option 130, 132, the process works stably and also for extreme variations of the scene.

[0168] According to other embodiments, the first scene abstraction is characterized by the BD-IR image data, by the SS-BD semantic segmentation. For example, in other embodiments, providing 180 ([Fig. 10]) a sufficient amount of TD training data makes the process more robust for different types of vehicles, installation positions, camera properties, and lighting conditions; some embodiments utilize the fact that, for example, certain components have the same appearance in different vehicles and are located in analogous positions. This characteristic translates, in other embodiments, for example, in the 142 training ([Fig. 9]), into over-adaptation to this application case, which, in other embodiments, simultaneously reduces the volume of TD training data required.

[0169] Other examples of implementation group a second level of abstraction of the classes fixed by the SS-BD semantic segmentation to a predefined number, for example three categories (example: first type of foreground VG-1, second type of foreground VG-2, background HG), to deduce, by prediction, the orientation of the camera.

[0170] According to other embodiments, classes that are not decisive for making the decision for a given application are intentionally ignored. This avoids other embodiments that could lead to areas of uncertainty between the three categories with back-and-forth movement, which, where applicable, would render a decision unstable.

[0171] According to other embodiments, for example, in a first step, semantic segmentation is learned for the vehicle's interior. This means that, in other embodiments, the classification of the vehicle's occupants into classes is learned in a pixelated way, such as, for example, the front seat backrest, the rear seat backrest, the front seat cushion, the rear seat cushion, window, ceiling, center console, people, other objects, etc. (see [Fig. 4A], 4B). This applies to other embodiments, for example, for many different vehicles and different seating positions. In other embodiments, this allows the NN neural network for semantic segmentation to be able, for example, also for unknown vehicles and unknown seating positions, to achieve a sufficient result (e.g., semantic segmentation capability).The same remark applies to other examples of . Implementation for different types of cameras and lighting conditions. For cameras, these other examples of implementation include slightly different aperture angles, but also variations between color images and near-infrared (NIR) images.

[0172] According to other embodiments, shooting at different times of day and / or different lighting or shadow conditions in the TD training data will result in robust training of the NN neural network.

[0173] According to other examples, the process is robust and can work for a very large number of poorly classified image points, for example, if a majority decision is applied.

[0174] According to other embodiments, the method applies to in-vehicle control, i.e. monitoring, of the vehicle's 10-IR passenger compartment.

Claims

Claims

1. Method, for example, computer-implemented method for processing image data (BD-IR) associated with the passenger compartment (10-IR) of a vehicle, for example, a motor vehicle (10) by a camera (1) comprising: - providing (100) a semantic segmentation (SS-BD) of the image data (BD-IR), - determining (102) at least one first region associated with the image data (BD-IR), for example, a foreground region (REG-1) based on the semantic segmentation (SS-BD), - determining (104) at least one of the following based at least on the first region (REG-1): a) position (1-POS) of the camera (1) in the passenger compartment (10-IR) of the vehicle, b) orientation (1-OR) the camera (1) in the passenger compartment (10-IR) of the vehicle.

2. The method of claim 1, wherein providing (100) the semantic segmentation (SS-BD) of at least one of the following comprises: a) determining (100a) the semantic segmentation (SS-BD), for example, using a neural network (NN), b) receiving (100b) the semantic segmentation (SS-BD), for example, from another unit, c) calling (100c) the semantic segmentation (SS-BD), for example, from another unit.

3. Method according to at least one of the preceding claims, comprising: - using (106) the position (1-POS) and / or the orientation (1-OR) of the camera (1) for at least one of the following steps: a) exploiting (AW) the image data (BD-IR), b) checking (UEB) the installation location and / or the installation position of the camera (1) in the passenger compartment (10-IR) of the vehicle, c) calibrating (KAL) the camera (1) or a system or method using the data (BD-IR) of the camera (1).

4. Method according to at least one of the preceding claims, consisting of: - determining (110) the first region (REG-1) associated with the image data (BD-IR) and at least one other region (REG-2) associated with the image data (BD-IR) based on the semantic segmentation (SS-BD), - determining (112) the position (1-POS) of the camera (1) and / or the orientation (1-OR) of the camera (1) in the passenger compartment (10-IR) of the vehicle based on the first region (REG-1) associated with the image data (BD-IR) and at least one other region (REG-2) associated with the image data (BD-IR).

5. method according to claim 4, according to which the first region (REG-1) or at least one other region (REG-2) characterizes one of the following elements: a) foreground region (VG-1, VG-2), b) background region (HG), * for example the foreground region (VG 1-VG-2) comprises at least one of the following elements: a) ceiling (Bl) of the passenger compartment (10-IR) of the vehicle, b) seat area (B2, B2'), for example, the background region (HG) not comprising the ceiling(s) of the passenger compartment of the vehicle (10-IR), or one or the other seat area.

6. Method according to at least one of the preceding claims, comprising: - carrying out (120) a classification (KLASS), for example, by image points, of image data (BD-IR) into several classes, * for example, several classes characterizing at least one of the following elements of the passenger compartment (10-IR) of the vehicle, a) the front seat backrest, b) the rear seat backrest, c) the front seat cushion, d) the rear seat cushion, e) a window, f) the ceiling, g) the center console, h) people i) other objects and optionally, using (122) the classification (KLASS), for example, to determine (102) at least a first region (REG-1) associated with the image data (BD-IR).

7. Method according to claim 6, comprising: - associating (130) the image points of a first group (GR-KL-1) of classes with the first region (REG-1) and optionally, - associating (132) the image points of a second group (GR-KL-2) of classes with one or more second regions (REG-2), * for example, the classes of the second group (GR-KL-2) being different from the classes of the first group (GR-KL-1) and optionally, - consolidating (134) the first region (REG-1) or the second region (REG-2).

8. Method according to claim 7, comprising: - transforming (135) the image data (BE-REG-12) characterizing the first region (REG-1) and the second region (REG-2) into a vector, for example, a column vector (SV) whose elements respectively characterize a dominant region of a corresponding row of the image data (BE-REG-12) characterizing the first region (REG-1) and the second region (REG-2), * for example, transforming (135) consists of executing (135a) a majority decision on the image data of a respective row of image data (BE-REG-12) characterizing the first region (REG-1) and the second region (REG-2), - using (137) the column vector (SV) for at least one of the following steps: a) determining (104) the position of the camera (1) in the passenger compartment (10-IR) of the vehicle, b) determining (104) the orientation (1-OR), for example, the vertical orientation of the camera in the passenger compartment (10-IR) of the vehicle.

9. Method according to at least one of the preceding claims, consisting of at least one of the following elements: - providing (140) a neural network (NN), - training (142) the neural network (NN), - determining (144) the semantic segmentation (SS-BD) of the image data (BD-IR) with the neural network (NN).

10. A method, for example, a computer-implemented method for training a neural network (NN) for semantic segmentation of image data of a camera (1) comprising the steps of: - providing (180) training data (TD) which comprises image data associated with a passenger compartment (10-IR) of at least one vehicle, for example, a motor vehicle, - training (182) the neural network (NN) for semantic segmentation (SS-BD) of image data associated with a passenger compartment (10-IR) of a vehicle, for example, a motor vehicle (10) and optionally, - using (184) the neural network (NN) to determine the semantic segmentation (SS-BD) of image data (BD-IR) of the camera (1).

11. The method of claim 10, wherein training (182) comprises at least one of the following steps: a) taking into account (182a) different vehicles or vehicle types, b) taking into account (182b) different installation positions or orientations of the camera (1), c) taking into account (182c) different types of cameras or different lighting conditions.

12. Device (200) for applying the method according to at least one of the preceding claims.

13. Computer-readable memory medium (SM) comprising instructions (PRG) whose execution by a computer (202) causes it to apply the method according to at least one of claims 1 to 11.

14. Computer program (PRG) comprising instructions which, by execution of the program (PRG) by a computer (202) control it to execute the method according to at least one of claims 1 to 11.

15. Data carrier signal (DCS) which the computer program (PRG) according to claim 14 transmits and / or characterizes.

16. Use (300) of the method according to at least one of claims 1 to 11 or of the device (10; 200) according to claim 12 or of the computer-readable storage medium (SM) according to claim 13 or of the computer program (PRG) according to claim 14 or the data carrier signal (DCS) according to claim 15 for at least one of the following steps: a) determining (301) the position (1-POS) of a camera (1) in the vehicle interior (10-IR), b) determining (302) the orientation (1-OR) of one or more cameras (1) in the passenger compartment (10-IR) of the vehicle, c) exploiting (303) the image data from the camera (1), d) checking (304) the installation location or installation position of the camera (1) in the passenger compartment (10-IR) of the vehicle, e) calibrating (305) the camera (1) or the system or method used for the image data (BD-IR) of the camera (1), f) increasing (306) robustness to a change of camera type, g) abstracting (307) a scene in the passenger compartment (10-IR) of the vehicle.