Method and device for processing image data associated with an interior of a vehicle
The method employs semantic segmentation and neural networks to classify and vectorize image data from vehicle interiors, addressing the challenge of camera alignment and calibration across varying conditions, ensuring accurate and robust camera positioning and orientation determination.
Patent Information
- Application Number
- PCT/EP2024/063906
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-21
- Publication Date
- 2024-12-05
AI Technical Summary
Existing methods for processing image data from vehicle interiors struggle to accurately determine the position and orientation of cameras, especially in varying vehicle types and lighting conditions, leading to difficulties in ensuring proper alignment and calibration.
A computer-implemented method using semantic segmentation, involving neural networks, to classify image data into regions such as foreground and background, and transform these regions into vectors for determining camera position and orientation, which allows for checking camera alignment and calibration within the vehicle interior.
The method effectively ensures proper camera alignment and calibration across different vehicle types and lighting conditions, enhancing robustness against changes in camera type and scene variations, and enabling reliable in-vehicle monitoring.
Smart Images

Figure EP2024063906_05122024_PF_FP_ABST
Abstract
Description
[0001] Description
[0002] Method and apparatus for processing image data associated with a vehicle interior of a vehicle
[0003] State of the art
[0004] The disclosure relates to a method for processing image data associated with a vehicle interior of a vehicle.
[0005] The disclosure further relates to an apparatus for processing image data associated with a vehicle interior of a vehicle.
[0006] Disclosure of the invention
[0007] Example embodiments relate to a method, for example a computer-implemented method, for processing image data from a camera associated with a vehicle interior of a vehicle, for example a motor vehicle, comprising: providing a semantic segmentation of the image data, determining at least one first region associated with the image data, for example a foreground region, based on the semantic segmentation, determining at least one of the following elements based at least on the first region: a) position of the camera in the vehicle interior, b) orientation of the camera in the vehicle interior. In example embodiments, this enables, for example, a verification of a correct arrangement and / or alignment of the camera in the vehicle interior.
[0008] In exemplary embodiments, it is provided that providing the semantic segmentation comprises at least one of the following elements: a) determining the semantic segmentation, for example using a neural network, b) receiving the semantic segmentation, for example from another unit, c) retrieving the semantic segmentation, for example from another unit.
[0009] In exemplary embodiments, it is provided that the method comprises: using the position and / or the orientation of the camera for at least one of the following elements: a) evaluation of the image data, b) checking an installation location and / or an installation position of the camera in the vehicle interior, c) calibration of the camera and / or a system or method using the image data of the camera.
[0010] In exemplary embodiments, the method comprises: determining the first region associated with the image data and at least one further region associated with the image data based on the semantic segmentation, determining the position of the camera and / or the orientation of the camera in the vehicle interior based on the first region associated with the image data and the at least one further region associated with the image data.
[0011] In exemplary embodiments, it is provided that the first region or the at least one further region characterizes one of the following elements: a) foreground region, b) background region, wherein, for example, the foreground region has at least one of the following elements: a) ceiling of the vehicle interior, b) seating area, wherein, for example, the background region has neither a or the ceiling of the vehicle interior nor a or the seating area.
[0012] In exemplary embodiments, the method comprises: carrying out a classification, for example pixel-by-pixel, of the image data into a plurality of classes, wherein, for example, the plurality of classes characterize at least one of the following elements of the vehicle interior: a) front seat backrest, b) rear seat backrest, c) front seat surface, d) rear seat surface, e) window, f) ceiling, g) center console, h) people, i) other objects, and, optionally, using the classification, for example for determining the at least one first region associated with the image data. In exemplary embodiments, the method comprises: assigning pixels of a first group of classes to the first region, and, optionally, assigning pixels of a second group of classes to one or morethe second region, for example, wherein the classes of the second group are different from the classes of the first group, and, optionally, consolidating the first region and / or the second region.
[0013] In exemplary embodiments, the method comprises: transforming image data characterizing the first region and the second region into a vector, for example a column vector, whose elements each characterize a dominant region of a corresponding row of the image data characterizing the first region and the second region, wherein, for example, the transformation comprises carrying out a majority decision on image data of a respective row of the image data characterizing the first region and the second region, and, optionally, using the column vector for at least one of the following elements: a) determining the position of the camera in the vehicle interior, b) determining the orientation, for example vertical orientation, of the camera in the vehicle interior.
[0014] In exemplary embodiments, it is provided that the method comprises at least one of the following elements: providing a neural network, training the neural network, determining the semantic segmentation of the image data by means of the neural network.
[0015] Further exemplary embodiments relate to a method, for example a computer-implemented method, for training a neural network, for example an artificial, for example deep, neural network, for the semantic segmentation of image data from a camera, comprising the steps of: providing training data which comprise image data associated with a vehicle interior of at least one vehicle, for example a motor vehicle, training the neural network for a semantic segmentation of image data associated with a vehicle interior of a vehicle, for example a motor vehicle, and, optionally, using the neural network to determine the semantic segmentation of image data from the camera.In exemplary embodiments, it is provided that the training comprises at least one of the following elements: a) taking into account different vehicles and / or vehicle types, b) taking into account different installation positions and / or orientations of the camera, c) taking into account different camera types and / or lighting conditions.
[0016] Further exemplary embodiments relate to an apparatus for carrying out the method according to the embodiments.
[0017] Further exemplary embodiments relate to a computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to the embodiments.
[0018] Further exemplary embodiments relate to a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the embodiments.
[0019] Further exemplary embodiments relate to a data carrier signal that transmits and / or characterizes the computer program according to the embodiments.
[0020] Further exemplary embodiments relate to a use of the method according to the embodiments and / or the device according to the embodiments and / or the computer-readable storage medium according to the embodiments and / or the computer program according to the embodiments and / or the data carrier signal according to the embodiments for at least one of the following elements: a) determining a position of a camera in the vehicle interior, b) determining an orientation of one or the cameras in the vehicle interior, c) evaluating image data from the camera, d) checking an installation location and / or an installation position of the camera in the vehicle interior, e) calibrating the camera and / or a system or method using the image data from the camera, f) increasing robustness against a change of camera type, g) abstracting a scene in the vehicle interior.
[0021] Further features, possible applications, and advantages of the invention will become apparent from the following description of exemplary embodiments of the invention, which are illustrated in the figures of the drawing. All described or illustrated features, individually or in any combination, constitute the subject matter of the invention, regardless of their summary in the claims or their references, as well as regardless of their wording or representation in the description or in the drawing.
[0022] The drawing shows:
[0023] Fig. 1 schematically shows a simplified flow diagram according to exemplary embodiments,
[0024] Fig. 2 schematically shows a simplified block diagram according to exemplary embodiments,
[0025] Fig. 3 schematically shows a simplified flow diagram according to exemplary embodiments,
[0026] Fig. 4A schematically illustrates aspects of semantic segmentation according to exemplary embodiments,
[0027] Fig. 4B schematically illustrates aspects of semantic segmentation according to exemplary embodiments,
[0028] Fig. 5A schematically illustrates aspects of regions according to exemplary embodiments,
[0029] Fig. 5B schematically illustrates aspects of regions according to exemplary embodiments, Fig. 6A schematically illustrates aspects of regions according to exemplary embodiments,
[0030] Fig. 6B schematically illustrates aspects of regions according to exemplary embodiments,
[0031] Fig. 6C schematically illustrates aspects of regions according to exemplary embodiments,
[0032] Fig. 7 schematically shows a simplified flow diagram according to exemplary embodiments,
[0033] Fig. 8A schematically shows a simplified flow diagram according to exemplary embodiments,
[0034] Fig. 8B schematically shows a simplified flow diagram according to exemplary embodiments,
[0035] Fig. 9 schematically shows a simplified block diagram according to exemplary embodiments,
[0036] Fig. 10 schematically shows a simplified flow diagram according to exemplary embodiments,
[0037] Fig. 11 schematically shows a simplified block diagram according to exemplary embodiments,
[0038] Fig. 12 schematically illustrates aspects of uses according to exemplary embodiments.
[0039] Exemplary embodiments, see Fig. 1, 2, relate to a method, for example a computer-implemented method, for processing image data BD-IR of a camera 1 associated with a vehicle interior 10-1 R of a vehicle, for example a motor vehicle, 10, comprising: providing 100 (Fig. 1) a semantic segmentation SS-BD of the image data BD-IR, determining 102 at least one first region REG-1 associated with the image data BD-IR, for example a foreground region, based on the semantic segmentation SS-BD, determining 104 at least one of the following elements based at least on the first region REG-1: a) position 1 -POS of the camera 1 in the vehicle interior 10-1 R, b) orientation 1 -OR of the camera 1 in the vehicle interior 10-1 R. In exemplary embodiments, this enables, for example, a check of a correct arrangement and / or alignment of the camera 1 in the vehicle interior 10-1 R.
[0040] In exemplary embodiments, Fig. 1 , it is provided that the provision 100 of the semantic segmentation SS-BD comprises at least one of the following elements: a) determining 100a the semantic segmentation SS-BD, for example using a neural network, b) receiving 100b the semantic segmentation SS-BD, for example from a further unit (not shown), c) retrieving 100c the semantic segmentation SS-BD, for example from a further unit.
[0041] In exemplary embodiments, Fig. 1 , it is provided that the method comprises: using 106 the position 1 -POS and / or the orientation 1 -OR of the camera 1 for at least one of the following elements: a) evaluation AW of the image data BD-IR, b) checking UEB of an installation location and / or an installation position of the camera 1 in the vehicle interior 10-1 R, c) calibration KAL of the camera 1 and / or of a system or method using the image data BD-IR of the camera 1.
[0042] In exemplary embodiments, Fig. 3, it is provided that the method comprises: determining 110 the first region REG-1 associated with the image data BD-IR (for example characterized by at least a part of the image data) and at least one further region REG-2 associated with the image data BD-IR (for example characterized by at least one, for example different, part of the image data) based on the semantic segmentation SS-BD (Fig. 1), determining 112 (Fig. 3) the position 1 -POS of the camera 1 and / or the orientation 1 -OR of the camera 1 in the vehicle interior 10-IR based on the first region REG-1 associated with the image data BD-IR and the at least one further region REG-1 associated with the image data BD-IR.In exemplary embodiments, it is provided that the first region REG-1 or the at least one further region REG-2 characterizes one of the following elements: a) foreground region, b) background region, wherein, for example, the foreground region has at least one of the following elements: a) ceiling of the vehicle interior, b) seating area, wherein, for example, the background region has neither a or the ceiling of the vehicle interior nor a or the seating area.
[0043] For example, the first region REG-1 can thus characterize a foreground region, which includes, for example, a ceiling of the vehicle interior and a seating area. Further details are described below, for example, with reference to Figs. 4A, 4B, 5A, 5B.
[0044] Fig. 4A schematically shows aspects of a semantic segmentation according to exemplary embodiments, such as can be obtained, for example, according to block 100 of Fig. 1 based on the image data BD-IR of the camera 1. For example, the image data BD-IR comprise one or more camera images or (e.g., by means of image processing) images of the interior 10-1 R (Fig. 2) derived therefrom. In the exemplary semantic segmentation, as depicted in Fig. 4A, each pixel is assigned to one of several possible classes, thus, for example, there is a pixel-by-pixel classification of a camera image. In other words, the aforementioned semantic segmentation can also be characterized as an image, wherein the values assigned to the individual pixels characterize the respective class based on the aforementioned classification. For example, the pixels in the area B1 according to Fig. 4A are assigned to a first part (e.g.,a ceiling) of the interior 10-1 R, the pixels in the area or areas B2 are assigned to a second, different, part (e.g. seats) of the interior 10-1 R, etc.
[0045] Fig. 4B schematically shows aspects of a semantic segmentation, similar to Fig. 4A, but based on a different, for example vertical, orientation of the camera 1. From Fig. 4B it can be seen that the image areas B2 associated with the seats are present, but not the image area B1 associated with the ceiling. In exemplary embodiments, Fig. 7, it is provided that the method comprises: carrying out 120 a, for example pixel-wise, classification KLASS of the image data BD-IR into a plurality of classes, wherein, for example, the plurality of classes characterize at least one of the following elements of the vehicle interior 10-1 R: a) front seat backrest, b) rear seat backrest, c) front seat seat surface, d) rear seat seat surface, e) window, f) ceiling (see, for example, also the area B1 according to Fig.4A), g) center console, h) persons, i) other objects, and, optionally, using 122 the classification KLASS, for example for determining 102 the at least one first region REG-1, REG-2 associated with the image data BD-IR.
[0046] In exemplary embodiments, Fig. 8A, it is provided that the method comprises: assigning 130 pixels of a first group GR-KL-1 of classes to the first region REG-1, and, optionally, assigning 132 pixels of a second group GR-KL-2 of classes to one or the second region REG-2, wherein, for example, the classes of the second group are different from the classes of the first group, and, optionally, consolidating 134 (e.g., uniting sub-areas) the first region REG-1 and / or the second region REG-1.
[0047] Fig. 5A shows, by way of example, the first region REG-1, specifically comprising not necessarily contiguous areas B1, B2, which characterize, for example, different foreground areas VG-1, VG-2. The first region REG-1 according to Fig. 5A can be obtained, for example, based on the assignment 130 according to Fig. 8A. For example, in some exemplary embodiments, the first region REG-1 can be assigned those classes of pixels that are characterized in Fig. 5A by the reference symbols B1 and B2, i.e., for example, pixels that belong to seat surfaces or seats, pixels that belong to the steering wheel and a dashboard cover, pixels that belong to the ceiling, etc.For example, in some exemplary embodiments, all other classes of pixels can be assigned to the second region REG-2, which characterizes a background HG, for example, i.e., contains pixels that have not been assigned to the first region REG-1. Fig. 5B shows, by way of example, consolidated areas B1', B2' or regions REG-1', REG-2' that can be obtained by consolidating 134 according to Fig. 8A, which, for example, still allow a distinction between foreground VG-1, VG-2 (region REG-1') and background HG', REG-2', which is sufficient for some evaluations or applications in further exemplary embodiments, but may reduce the complexity compared to the representation in Fig. 5A.
[0048] In exemplary embodiments, Fig. 8B, it is provided that the method comprises: transforming 135 image data characterizing the first region REG-1 and the second region REG-2 into a vector, for example a column vector SV, whose elements each characterize a dominant region of a corresponding row of the image data characterizing the first region REG-1 and the second region REG-2, wherein, for example, the transforming 135 comprises carrying out 135a a majority decision on image data of a respective row of the image data characterizing the first region REG-1 and the second region REG-2, and, optionally, using 137 the column vector SV for at least one of the following elements: a) determining the position 1 -POS of the camera 1 in the vehicle interior 10-1 R, b) determining the orientation 1 -OR, for example vertical orientation, of the camera 1 in the vehicle interior 10-IR.
[0049] Fig. 6A shows, by way of example, an image similar to Fig. 5B, with consolidated regions REG-1', REG-2' which are assigned to a respective foreground VG-1, VG-2 or background HG. For the transformation 135 according to Fig. 8B, an element of the aforementioned column vector SV can be determined for each row of the image characterized by Fig. 6A, e.g. by majority vote. For this purpose, it is determined in a respective row Z of the image characterized by Fig. 6A to which region REG-1, REG-2 or to which region VG-1, VG-2, HG' the most pixels of the considered row Z belong, and this provides the value of the element E of the column vector SV.
[0050] In exemplary embodiments, the column vector SV can be evaluated, for example, by determining neighboring vector elements of different values, whose corresponding rows are thus assigned, for example, to different regions, see the double arrows a1, a2, or their position in the column vector SV. On this basis, in further exemplary embodiments, for example, a vertical orientation of the camera 1 (Fig. 2) in the interior space 10-1 R can be determined.
[0051] Fig. 6B shows, by way of example, an image similar to Fig. 6A, wherein in the image according to Fig. 6B, however, a foreground region VG-1 associated with the ceiling of the interior 10-1 R occupies a comparatively large portion of the image. This accordingly leads to a column vector (not labeled in Fig. 6B) with a comparatively large number of elements corresponding to the foreground region VG-1. Based on the situation according to Fig. 6B or the corresponding column vector, it can be concluded, for example, that camera 1 (Fig. 2) is vertically misaligned, e.g. because it contains too large a proportion of pixels associated with the ceiling.
[0052] Fig. 6C shows an example of a mapping similar to Fig. 6A, whereby the mapping according to Fig. 6C provides a comparatively balanced representation of the different regions VG-1, VG-2, HG, and REG-1', REG-2', respectively. This accordingly leads to a column vector not labeled in Fig. 6C.
[0053] In further exemplary embodiments, the semantic segmentation SS-BD (Fig. 1) can thus be used, for example, to identify foreground regions VG-1, VG-2 (Figs. 5A, 5B), which, for example, contain the ceiling on one side and the seating areas on the other side. A background region HG is defined, for example, as a region that belongs neither to the ceiling nor to the seating region.
[0054] As described above by way of example with reference to Fig. 6A, in further exemplary embodiments, for a division into the different image regions REG-1, REG-2, for example in Fig. 6A, for example from left to right, a respective class type with the highest number of pixels can be determined, which belong either to the classes of the ceiling, the seating areas or the background region. Subsequently, in further exemplary embodiments, for example, a boundary is determined, for example between the ceiling region VG-1 and the seating area region VG-2, see for example the double arrows a1, a2 according to Fig. 6A. In some exemplary embodiments, the boundary is located, for example, at least approximately in the middle of the two regions and is bordered, for example, by the background region HG'. In further exemplary embodiments, for example, if the background region HG' should be too large (e.g. relatively speaking), this can be an indication that, for exampleeither the seating region or the ceiling region was obscured. In this case, in further exemplary embodiments, the boundary can be adapted, for example, to the foreground region that is more strongly represented in the image and, for example, drawn with a predefined offset (d), see Fig. 6C.
[0055] In exemplary embodiments, Fig. 9, it is provided that the method comprises at least one of the following elements: providing 140 a or the neural network NN, training 142 the neural network NN, determining 144 the semantic segmentation SS-BD of the image data by means of the neural network NN.
[0056] Further exemplary embodiments, Fig. 10, relate to a method, for example a computer-implemented method, for training a neural network NN, for example an artificial, for example deep, neural network, for the semantic segmentation of image data BD-IR of a camera 1, comprising the steps: providing 180 training data TD which has image data BD-IR associated with a vehicle interior 10-1 R of at least one vehicle 10, for example a motor vehicle, training 182 the neural network NN for a semantic segmentation of image data associated with a vehicle interior of a vehicle, for example a motor vehicle, and, optionally, using 184 the (e.g. trained) neural network NN to determine the semantic segmentation SS-BD of image data of the camera.
[0057] In exemplary embodiments, Fig. 10, it is provided that the training 182 comprises at least one of the following elements: a) taking into account 182a different vehicles and / or vehicle types, b) taking into account 182b different installation positions and / or orientations of the camera 1, c) taking into account 182c different camera types and / or lighting conditions. Further exemplary embodiments, Fig. 11, relate to a device 200 for carrying out the method according to the embodiments.
[0058] In further exemplary embodiments, Fig. 11 , it is provided that the device 200 comprises: a computing device ("computer") 202 having at least one computing core, a memory device 204 assigned to the computing device 202 for at least temporarily storing at least one of the following elements: a) data DAT (e.g. the image data BD-IR or data derivable therefrom, e.g. data characterizing the semantic segmentation SS-BD or the semantic segmentation SS-BD, determined classes, determined regions, etc.), b) computer program PRG, for example for carrying out the method according to the embodiments.
[0059] In further exemplary embodiments, the memory device 204 comprises a volatile memory (e.g., random access memory (RAM)) 204a, and / or a non-volatile (NVM) memory (e.g., flash EEPROM) 204b, or a combination thereof or with other memory types not explicitly mentioned.
[0060] Further exemplary embodiments relate to a computer-readable storage medium SM comprising instructions PRG' which, when executed by a computer 202, cause the computer to carry out the method according to the embodiments.
[0061] Further exemplary embodiments relate to a computer program PRG comprising instructions which, when the program PRG is executed by a computer 202, cause the computer 202 to carry out the method according to the embodiments.
[0062] Further exemplary embodiments relate to a data carrier signal DCS that characterizes and / or transmits the computer program PRG according to the embodiments. The data carrier signal DCS can be received, for example, via an optional data interface 206 of the device 200.
[0063] Further exemplary embodiments, Fig. 12, relate to a
[0064] Use 300 of the method according to the embodiments and / or the device 200 according to the embodiments and / or the computer-readable storage medium SM according to the embodiments and / or the computer program PRG, PRG' according to the embodiments and / or the data carrier signal DCS according to the embodiments for at least one of the following elements: a) determining 301 a position of a camera 1 in the vehicle interior 10-1 R (Fig. 2), b) determining 302 an orientation of one or the cameras 1 in the vehicle interior 10-1 R, c) evaluating 303 image data BD-IR of the camera 1, d) checking 304 an installation location and / or an installation position of the camera 1 in the vehicle interior 10-1 R, e) calibrating 305 the camera 1 and / or a system (not shown) or method using the image data of the camera, f) increasing 306 a robustness against a change of a camera type, g) Abstracting 307 a scene in the vehicle interior 10-1 R.
[0065] Further exemplary aspects and embodiments are described below, which in further exemplary embodiments can each be combined individually or in any combination with at least one of the above aspects.
[0066] In exemplary embodiments, the principle according to the embodiments can be used, for example, for checking a camera position in any vehicle interior. In exemplary embodiments, a deep convolutional neural network, e.g., of the type described in the publication "ImageNet Classification with Deep Convolutional Neural Networks" by A. Krizhevsky, NIPS2021, can be used for semantic segmentation 100 (Fig. 1).
[0067] In exemplary embodiments, a correct orientation of the camera 1, e.g. for any vehicle type, is derived based on an output of the neural network NN.
[0068] Exemplary embodiments can thus, for example, contribute to solving the problem that, with a variable installation position of a camera 1 in any vehicle 10 (Fig. 2) or vehicle interior, calibration of the camera 1 can be comparatively difficult and, as a result, with conventional approaches, for example, the correct alignment of the camera cannot be ensured.
[0069] In further exemplary embodiments, a vehicle interior 10-IR differs considerably between different vehicles or vehicle types, sometimes resulting in even greater variation, for example, due to an unknown installation position. The principle according to the embodiments can predict the correct installation position despite the aforementioned possible variations.
[0070] In further exemplary embodiments, the principle according to the embodiments is comparatively robust, e.g. against changes in camera type, which has the advantage, e.g. when the optical elements of the camera are changed, that a method for evaluating the camera images does not have to be readjusted.
[0071] In further exemplary embodiments, the method according to the embodiments also functions, for example, during a journey, e.g., when the seats are partially or fully occupied. This facilitates calibration, for example, and also enables the function to be used while driving.
[0072] In further exemplary embodiments, an abstraction of the scene of the camera image by means of the semantic segmentation SS-BD (Fig. 1) makes the method, for example, also robust against light changes and can be used, for example, both during the day and at night, e.g. with an NIR (near-infrared) camera.
[0073] In further exemplary embodiments, a scene in the interior 10-IR is abstracted to such an extent, for example by the semantic segmentation and the optional assignment 130, 132, that the method functions stably, for example, even in the case of extreme variation of the scene.
[0074] In further exemplary embodiments, a first abstraction of the scene, as characterized by the image data BD-IR, is carried out by the semantic segmentation SS-BD. For example, in further exemplary embodiments, by providing 180 (Fig. 10) a sufficient amount of training data TD, the method can be made robust with respect to different vehicle types, installation positions, camera properties, and lighting conditions. In exemplary embodiments, the fact that, for example, certain components between different vehicles look similar and are located in similar positions is used. In further exemplary embodiments, this feature leads, for example, during training 142 (Fig. 9), to a desired overfitting to this use case, which in further exemplary embodiments can, for example, simultaneously reduce the amount of required training data TD.
[0075] In further exemplary embodiments, a second abstraction level groups the classes output by the semantic segmentation SS-BD into a predeterminable number of, e.g., three, categories (e.g., first foreground type VG-1, second foreground type VG-2, background HG), e.g., in order to predict the camera direction therefrom.
[0076] In further exemplary embodiments, classes that are not relevant for decision-making, e.g., in a given application, are deliberately ignored. This prevents, for example, uncertain areas from switching back and forth between the three categories, which could potentially render a decision unstable.
[0077] In further exemplary embodiments, for example in a first step, the aforementioned semantic segmentation for the vehicle interior is learned. In further exemplary embodiments, this means, for example, that a pixel-by-pixel classification of the vehicle interior into classes such as front seat backrest, rear seat backrest, front seat seat surface, rear seat seat surface, window, ceiling, center console, people, other objects, etc. is learned, see Fig. 4A, 4B. In further exemplary embodiments, this occurs, for example, on the basis of many different vehicles and different installation positions. In further exemplary embodiments, this enables the neural network NN capable of semantic segmentation to deliver sufficient performance (e.g. performance with regard to semantic segmentation), for example even for unknown vehicles and unknown installation positions.The same applies to other exemplary embodiments for different camera types and / or lighting conditions. For cameras, this would include slightly different aperture angles in other exemplary embodiments, but also a variation between color and near-infrared (NIR) images.
[0078] In further exemplary embodiments, the presence of images taken at different times of day and / or different light and shadow conditions in the training data TD contributes to a robust trained neural network NN.
[0079] In further exemplary embodiments, the method according to the embodiments is so robust that it functions reliably even with a relatively large number of incorrectly classified pixels, e.g. when the described majority decision is carried out.
[0080] In further exemplary embodiments, the method according to the embodiments can also be used, for example, for in-vehicle monitoring, i.e. monitoring, for example, the interior 10-1 R.
Claims
Claims 1 . Method, for example a computer-implemented method, for processing image data (BD-IR) of a camera (1) associated with a vehicle interior (10-1 R) of a vehicle, for example a motor vehicle (10), comprising: providing (100) a semantic segmentation (SS-BD) of the image data (BD-IR), determining (102) at least one first region associated with the image data (BD-IR), for example a foreground region (REG-1) based on the semantic segmentation (SS-BD), determining (104) at least one of the following elements based at least on the first region (REG-1): a) position (1 -POS) of the camera (1) in the vehicle interior (10-1 R), b) orientation (1 -OR) of the camera (1) in the vehicle interior (10-IR).
2. The method according to claim 1, wherein providing (100) the semantic segmentation (SS-BD) comprises at least one of the following elements: a) determining (100a) the semantic segmentation (SS-BD), for example using a neural network (NN), b) receiving (100b) the semantic segmentation (SS-BD), for example from a further unit, c) retrieving (100c) the semantic segmentation (SS-BD), for example from a further unit.
3. Method according to at least one of the preceding claims, comprising: using (106) the position (1 -POS) and / or the orientation (1 -OR) of the camera (1) for at least one of the following elements: a) evaluation (AW) of the image data (BD-IR), b) checking (UEB) an installation location and / or an installation position of the camera (1) in the vehicle interior (10-IR), c) calibration (KAL) of the camera (1) and / or a system or method using the image data (BD-IR) of the camera (1).
4. The method according to at least one of the preceding claims, comprising: determining (110) the first region (REG-1) associated with the image data (BD-IR) and at least one further region (REG-2) associated with the image data (BD-IR) based on the semantic segmentation (SS-BD), determining (112) the position (1 -POS) of the camera (1) and / or the orientation (1 -OR) of the camera (1) in the vehicle interior (10-1 R) based on the first region (REG-1) associated with the image data (BD-IR) and the at least one further region (REG-2) associated with the image data (BD-IR).
5. The method according to claim 4, wherein the first region (REG-1) or the at least one further region (REG-2) characterizes one of the following elements: a) foreground region (VG-1, VG-2), b) background region (HG), wherein, for example, the foreground region (VG-1, VG-2) has at least one of the following elements: a) ceiling (B1) of the vehicle interior (10-1 R), b) seating area (B2, B2'), wherein, for example, the background region (HG) has neither a or the ceiling of the vehicle interior (10-IR) nor a or the seating area.
6. The method according to at least one of the preceding claims, comprising: carrying out (120) a classification (KLASS), for example pixel-wise, of the image data (BD-IR) into a plurality of classes, wherein, for example, the plurality of classes characterize at least one of the following elements of the vehicle interior (10-IR): a) front seat backrest, b) rear seat backrest, c) front seat seat surface, d) rear seat seat surface, e) window, f) ceiling, g) center console, h) people, i) other objects, and, optionally, using (122) the classification (KLASS), for example for determining (102) the at least one first region (REG-1) associated with the image data (BD-IR).
7. The method according to claim 6, comprising: assigning (130) pixels of a first group (GR-KL-1) of classes to the first region (REG-1), and, optionally, assigning (132) pixels of a second group (GR-KL-2) of classes to one or the second region (REG-2), wherein, for example, the classes of the second group (GR-KL-2) are different from the classes of the first group (GR-KL-1), and, optionally, consolidating (134) the first region (REG-1) and / or the second region (REG-2).
8. The method according to claim 7, comprising: transforming (135) image data (BE-REG12) characterizing the first region (REG-1) and the second region (REG-2) into a vector, for example a column vector (SV), whose elements each characterize a dominant region of a corresponding row of the image data (BE-REG12) characterizing the first region (REG-1) and the second region (REG-2), wherein, for example, the transforming (135) comprises carrying out (135a) a majority decision on image data of a respective row of the image data (BE-REG12) characterizing the first region (REG-1) and the second region (REG-2), and, optionally, using (137) the column vector (SV) for at least one of the following elements: a) determining (104) the position (1 -POS) of the camera (1) in the vehicle interior (10-1 R), b) determining (104) the orientation (1 -OR), for example vertical orientation, of the camera (1) in the vehicle interior (10-1 R).
9. Method according to at least one of the preceding claims, comprising at least one of the following elements: providing (140) a or the neural network (NN), training (142) the neural network (NN), determining (144) the semantic segmentation (SS-BD) of the image data (BD-IR) by means of the neural network (NN).
10. Method, for example a computer-implemented method, for training a neural network (NN) for the semantic segmentation of image data from a camera (1), comprising the steps of: providing (180) training data (TD) which comprise image data associated with a vehicle interior (10-1 R) of at least one vehicle, for example a motor vehicle, training (182) the neural network (NN) for a semantic segmentation (SS-BD) of image data associated with a vehicle interior (10-1 R) of a vehicle, for example a motor vehicle (10), and, optionally, using (184) the neural network (NN) to determine the semantic segmentation (SS-BD) of image data (BD-IR) from the camera (1).
11. The method of claim 10, wherein the training (182) comprises at least one of the following elements: a) taking into account (182a) different vehicles and / or vehicle types, b) taking into account (182b) different installation positions and / or orientations of the camera (1), c) taking into account (182c) different camera types and / or lighting conditions.
12. Device (200) for carrying out the method according to at least one of the preceding claims.
13. Computer-readable storage medium (SM), comprising instructions (PRG) which, when executed by a computer (202), cause the computer to carry out the method according to at least one of claims 1 to 11.
14. A computer program (PRG) comprising instructions which, when the program (PRG) is executed by a computer (202), cause the computer (202) to carry out the method according to at least one of claims 1 to 11.
15. Data carrier signal (DCS) that transmits and / or characterizes the computer program (PRG) according to claim 14.
16. Use (300) of the method according to at least one of claims 1 to 11 and / or the device (10; 200) according to claim 12 and / or the computer-readable storage medium (SM) according to claim 13 and / or the computer program (PRG) according to claim 14 and / or the data carrier signal (DCS) according to claim 15 for at least one of the following elements: a) determining (301) a position (1 -POS) of a camera (1) in the vehicle interior (10-1 R), b) determining (302) an orientation (1 -OR) of one or more cameras (1) in the vehicle interior (10-IR), c) evaluating (303) image data of the camera (1), d) checking (304) an installation location and / or an installation position of the camera (1) in the vehicle interior (10-IR), e) calibrating (305) the camera (1) and / or one of the image data (BD-IR) of the camera (1) using the system or method f) increasing (306) a robustness against a change of a camera type, g) abstracting (307) a scene in the vehicle interior (10-IR).
Citation Information
Patent Citations
Departure warning method and system of mounting angle of elevator camera
CN110316630A
Computer-implemented method and system to support the installation of an imaging sensor and training procedure
DE102021206625A1
