Method for checking whether at least one object is trapped in a closed vehicle door of a motor vehicle

Abstract

The invention relates to a method for checking whether at least one object (4) is trapped in a closed vehicle door (2) of a motor vehicle (1), for the motor vehicle (1).The method comprises: receiving at least one camera image (12, 13, 14, 15) that describes at least part of the environment (10) of the motor vehicle (1); determining a bird's-eye view image (16) of the environment (10) of the motor vehicle (1), wherein the bird's-eye view image (16) describes at least part of at least one outer wall (3) of the motor vehicle (1), by applying a bird's-eye view image determination algorithm (17) to the at least one received camera image (12, 13, 14, 15); checking whether the at least one object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1), by applying an object recognition algorithm (20) to the determined bird's-eye view image (16); and if it is determined that this is the case, obtain object information (19) that at least describes that at least one object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1).

Description

The invention relates to a method for checking whether at least one object is trapped in a closed vehicle door of a motor vehicle. The method is designed for a motor vehicle. The invention also relates to a control device, a motor vehicle, and a computer program for carrying out such a method. When a person enters a motor vehicle, they may trap at least one object in the door when it closes. This object could be, for example, part of the person's clothing, such as an abaya, skirt, dress, coat, scarf, or other textile. If the vehicle starts moving or continues driving with the object trapped, it could contribute to injuries and / or accidents. Therefore, the vehicle should be equipped with a system to detect objects trapped in closed doors. US patent 2023 / 0339401 A1 discloses a device for a vehicle comprising a camera and a processor. The processor is designed to evaluate a side-view image of the vehicle captured by the camera in order to detect a dangerous situation related to a vehicle door. The object of the invention is to provide a solution by means of which an object trapped in a closed vehicle door of a motor vehicle can be reliably detected. The problem is solved by the subject matter of the independent patent claims. A first aspect of the invention relates to a method for checking whether at least one object is trapped in a closed vehicle door of a motor vehicle. The method is intended for use in a motor vehicle; that is, in a preferred example, it is carried out in the motor vehicle. For example, the method can be carried out by a control device of the motor vehicle. The method can be understood, in particular, as a computer-implemented method. The object is, for example, a part of a garment, such as an abaya, a skirt, a dress, a coat, a scarf, or another textile. A closed vehicle door is a door of a motor vehicle that is in a closed state in which at least entering or exiting the vehicle is not possible. The procedure may include checking whether at least one vehicle door, and in particular all vehicle doors, is / are closed. Specifically, the procedure is only carried out if at least one vehicle door, and in particular all vehicle doors, is / are closed; that is, the procedural steps described below are carried out. The at least one object can, for example, protrude from an edge of the vehicle door where the vehicle door abuts adjacent areas of an outer wall of the vehicle or another vehicle door. "Clamped" within the meaning of the invention means, for example, the crushing or clamping of the at least one object between the edge of the vehicle door and the adjacent area of ​​the outer wall or the other vehicle door. The method comprises receiving at least one camera image. The camera image depicts at least a portion of the vehicle's surroundings. The camera image was captured by a surround-view camera of the vehicle. If the method is performed, for example, by the control unit in the vehicle, it can receive the at least one camera image from the surround-view camera. The surround-view camera can be, for example, a side camera located in a side mirror and / or at another location on a side of the vehicle, a front camera, and / or a rear camera of the vehicle. The surround-view camera can be part of a surround-view camera system of the vehicle, which is configured to capture a 360-degree view of the vehicle's surroundings. The camera image is, for example, a two-dimensional image. In a preferred example, multiple camera images are received, each depicting a portion of the vehicle's surroundings.The parts of the environment described by the multiple camera images can overlap at least partially or be completely different from one another. In one example, the camera image, or at least one of the camera images in the case of multiple received camera images, can be a fisheye image. The procedure involves generating a bird's-eye view image of the vehicle's surroundings. This bird's-eye view image describes at least part of an exterior wall of the vehicle. For example, the bird's-eye view image describes at least the portion of the vehicle's surroundings that is depicted by the received camera image. Furthermore, the bird's-eye view image describes at least part of the vehicle itself; that is, at least a portion of at least one exterior wall of the vehicle lies within the detection range of the camera that captured the image. The received camera image, or at least one of the multiple received camera images, therefore describes at least a portion of at least one exterior wall of the vehicle. This portion of the exterior wall includes at least a portion of at least one of the vehicle's doors.The aerial view can also depict the vehicle itself, for example, in the form of a model or an animation. In a preferred example, at least all exterior walls of the vehicle are included in the aerial view. Each exterior wall is a wall that separates the vehicle from its surroundings and connects a front or front section of the vehicle with a rear or rear section. The bird's-eye view image is generated by applying a bird's-eye view image generation algorithm to the at least one received camera image. This algorithm comprises at least one rule and / or instruction, the execution of which, based on the received camera image, results in a projection into a bird's-eye view. Thus, the camera image is projected or transferred from the camera perspective of the camera that captured it into a bird's-eye view. If multiple camera images are received, they can be combined into a single, composite image. In this example, the generated bird's-eye view image is derived from this single composite image, meaning it is based on the combined multiple camera images.If camera images are received from the front camera, the rear camera and both side cameras, the bird's-eye view image can be a 360-degree image of the vehicle's surroundings as well as at least partially of the vehicle's exterior walls from a bird's-eye view. It is assumed that at least one camera image is always received, which at least partially depicts one of the vehicle's outer walls, including at least part of a closed vehicle door located thereon. In a preferred example, the acquired bird's-eye view image completely depicts at least the two opposing outer walls of the vehicle on which the vehicle doors are located. In another preferred example, the bird's-eye view image depicts all of the vehicle's doors, as well as a portion of the areas of the outer wall and the vehicle's surroundings adjacent to the vehicle doors.The idea here is that the vehicle doors are located on the outer wall of the motor vehicle, and therefore the trapped object is expected to be on the outer wall, for example, viewed vertically at the bottom edge of the vehicle door and / or viewed longitudinally adjacent to the vehicle door, for example in front of and / or behind the vehicle door, depending on where a hinge of the vehicle door is not located. The procedure involves checking whether at least one object is trapped in the closed vehicle door. This is done by applying an object recognition algorithm to the acquired bird's-eye view image. The object recognition algorithm includes at least one rule and / or regulation, the execution of which, based on the acquired bird's-eye view image, determines whether an object is identifiable as being trapped in the closed vehicle door or not. For this to be the case, the object must, for example, protrude or extend at least partially from the outer wall. For instance, the object could protrude or extend from an edge of the closed vehicle door towards the surroundings of the vehicle. If it is determined that at least one object is trapped in the closed vehicle door, object information is collected that at least describes the fact that at least one object is trapped in the closed vehicle door. This object information might, for example, describe that a trapped object has been detected. It may also include further information, such as details about the detected object and / or the closed vehicle door. This provides a method for detecting any object of any shape that is trapped in a closed vehicle door. The method utilizes the generated bird's-eye view image, as this allows for a representation of the outer wall that clearly shows or makes recognizable any objects protruding from or extending from it. Since the object trapped in the vehicle door is expected to protrude from it and / or differ in color from the vehicle's outer wall, the bird's-eye view image is particularly suitable for identifying the object without, for example, mistaking it for objects that are not actually in direct contact with the vehicle door. Such misinterpretations would otherwise occur due to the bird's-eye view not being intended.Therefore, this method can be used to reliably detect objects trapped in the closed vehicle door. One embodiment provides that if it is determined that no object is trapped in the closed vehicle door, object information is retrieved describing the fact that no object is trapped in the closed vehicle door. Thus, it can be provided that object information is retrieved not only when it is determined that at least one object is trapped in the closed vehicle door, but also whenever no object is trapped in the closed vehicle door, with the object information retrieved in this case clearly describing that no trapped object could be detected. This ensures that object information is always retrieved when the method is executed, regardless of whether an object is actually detected or not.In other words, the determined object information can at least distinguish between the information "object trapped detected" and "object trapped not detected". Further processing or evaluation of the object information may, for example, depend on whether the object information describes the trapped object or not. Since object information is always determined and provided, further use of the object information is simplified. Another embodiment provides that, if object information is detected indicating that at least one object is trapped in the closed vehicle door, a warning message is displayed in the vehicle indicating the trapped object. This warning message can be visual, audible, and / or haptic. In a preferred example, a message is displayed on a display device in the vehicle indicating the trapped object. The display device includes at least one screen. Alternatively or additionally, the warning message can be displayed audibly via a loudspeaker system in the vehicle, which includes at least one loudspeaker, for example, as a voice warning message.The haptic warning signal can, for example, cause the vehicle door in which the object is trapped to vibrate, at least locally. This door can also be visually highlighted, for instance, by an activated light and / or indicator light. Furthermore, the haptic warning signal can include, for example, a vibration of the vehicle's steering wheel to indicate an additional audible and / or visual warning. In other words, the vehicle's occupants are warned that an object has been detected in a vehicle door. Subsequently, for example, the object can be manually removed by reopening the vehicle door with the object trapped inside, and the door can then be closed again without the object being trapped. No further warning message will be issued, as no object is detected, allowing the vehicle to be driven or continue driving. The warning message is a particularly useful response to the object information, as it allows the vehicle's occupants to be informed quickly and intuitively after the object has been detected. Another embodiment provides that if object information is detected indicating that at least one object is trapped in the closed vehicle door, the vehicle is prevented from moving, at least temporarily. "Moving" here refers to starting or continuing to move the vehicle. If the vehicle's movement is prevented, at least temporarily, a drive system of the vehicle is, for example, temporarily blocked. It can be prevented, for instance, from starting a drive system of the vehicle, i.e., its engine, when a start-stop button is pressed. This prevents the vehicle from moving and thus stops it. A time window can be specified, limiting how long the vehicle's driving is prevented. For example, if an occupant, particularly the driver, does not react within this time window and remove the trapped object from the vehicle door, the driving restriction can be lifted or ended, allowing the vehicle to be driven despite the trapped object. Thus, the vehicle's driving function can be intervened if an object is detected in the door, reliably preventing potential injuries and accidents caused by the trapped object. In a preferred example, both the driving of the motor vehicle is prevented, at least temporarily, and the warning message is displayed in the motor vehicle. Furthermore, one embodiment provides that the object information describes at least the vehicle door in which the object is trapped. The vehicle door could be, for example, a driver's door, a passenger door, and / or a rear door of the vehicle. The respective rear door could be located on the driver's side or on the passenger side. If, for example, there are four vehicle doors in the vehicle, the object information could describe the vehicle door of the four in which the object is trapped. If, in an example, objects are trapped in several vehicle doors, multiple pieces of object information could be determined, in particular, one piece of object information for each trapped object. Alternatively, a single piece of determined object information could describe all vehicle doors affected by trapped objects.In the case of four vehicle doors, for example, five different pieces of object information can be determined: one piece of object information describing that no object is trapped, and four pieces of object information describing, for each of the four vehicle doors, that the object is trapped in that specific door. If the vehicle has more than four doors, even more different pieces of object information are possible. If the object information describes the vehicle door in which the object is trapped, the warning message in the vehicle can specify which vehicle door is affected by the trapped object, so that the vehicle occupants can specifically open this vehicle door to remove the object quickly and with minimal effort, i.e., without having to open several vehicle doors in search of the trapped object. In an alternative example, the object information can be extended to include the trunk door of a vehicle, so that the determined object information can be used to describe whether an object is trapped in the trunk door or not. In this example, it is assumed that the bird's-eye view image at least partially describes an exterior rear wall or back panel of the vehicle, so that the object trapped in the trunk door is described by the bird's-eye view image. Another embodiment provides that, after acquiring the bird's-eye view image, at least a section of the bird's-eye view image is determined in which at least one outer wall of the vehicle lies, at least partially. The object recognition algorithm is applied only to the determined section of the bird's-eye view image. This section can be understood as an area of ​​interest within the bird's-eye view image. Alternatively, the section can be referred to as a region of interest (ROI). Since the trapped object is expected to be on the outer wall and not, for example, in the front and / or rear area and / or in a part of the vehicle's surroundings that is a minimum distance away from the outer wall, the object detection algorithm does not always need to be applied to the entire bird's-eye view image. Instead, the relevant section can be determined, and the object detection algorithm applied only to this section. This speeds up the application of the object detection algorithm and reduces the computational effort required to perform the procedure, compared to considering the entire bird's-eye view image.This is particularly useful for bird's-eye view images that show a 360-degree view of the vehicle's surroundings, as such bird's-eye view images describe numerous parts of the environment and / or the outer walls of the vehicle where the object trapped in the vehicle door cannot possibly be located. Another embodiment involves a fixed, predefined section of the bird's-eye view image. For example, a fixed portion of the bird's-eye view image can be designated as the section. This fixed section then describes at least a part of the outer wall as well as areas of the surroundings directly adjacent to this part of the outer wall. The section can be defined by image pixels in the bird's-eye view image. These image pixels can, for example, be stored in the control device. This simplifies the execution of the method, as the section does not need to be determined laboriously but can be found directly, for example, using the stored image pixels in the bird's-eye view image. In an alternative example, it may be provided that the section is recalculated for each bird's-eye view image, for example depending on the received camera images.Another embodiment involves the object recognition algorithm comprising at least one neural network. In a preferred example, the object recognition algorithm is based on machine learning methods. This at least one neural network is, for example, a deep neural network. This allows the object recognition algorithm to be specifically trained to recognize typical objects in vehicle doors, such as textiles or, in particular, certain types of textiles. By using the neural network-based object recognition algorithm, a more reliable detection of the trapped object can be achieved compared to classical image processing methods.Because the trapped object can have any shape, there can be a wide variety of shapes, which can typically still be processed well by the neural network, so that ultimately the object recognition algorithm reliably recognizes the trapped object. Before the procedure was carried out, the neural network was trained to recognize the object based on the bird's-eye view image. During the training process, a training dataset is fed into a previously untrained neural network. The training dataset can, for example, comprise numerous bird's-eye view images, fisheye images, and / or camera images, in particular several thousand individual images. In each image of the training dataset, at least one object was manually marked. Furthermore, in one example, three-dimensional models for the at least one object in the images can be created and fed into the neural network during the training process. "Feeded in" here means passing or inputting the data into the neural network. During the training process, the neural network learns to recognize the object trapped in the car door based on the bird's-eye view images.The trained neural network is then used as an object recognition algorithm when carrying out the procedure. Another embodiment involves at least one neural network performing instance segmentation to identify the object pixel-by-pixel in the bird's-eye view image, particularly in at least one section. Instance segmentation differs from semantic segmentation. Semantic segmentation identifies an object of a predefined class within the image under consideration—in this case, the bird's-eye view image. In other words, semantic segmentation locates objects within the image and categorizes them according to predefined categories. Each pixel in the bird's-eye view image is assigned a class or category name. In contrast, instance segmentation also includes locating specific objects based on the relationship between their corresponding pixels.For example, if multiple objects of the same category (or class) are present in the bird's-eye view image, each of these objects is treated as a unique instance. This means that not only is the presence of each object within a specific category (or class) recognized, but multiple objects can also be distinguished from one another. This is particularly relevant for determining which vehicle door an object is trapped in. This can be facilitated by providing the localization, which is included in the instance segmentation, for each detected object. Ultimately, the bird's-eye view image can pinpoint the exact location of a trapped object, down to the pixel. Another embodiment involves a neural network for instance segmentation employing a YOLACT architecture. YOLACT stands for "You Only Look At Coefficients." It is a well-known architecture for instance segmentation that enables essentially real-time, pixel-accurate segmentation. This allows the network to search for the object immediately after receiving the camera image and generating the bird's-eye view, thus enabling the timely detection of an object trapped in the vehicle door, or its absence. Alternative instance segmentation architectures are possible; the use of the YOLACT architecture is merely a preferred example. Another embodiment provides for the reception of four camera images captured by four different surround-view cameras of the vehicle. These four different surround-view cameras are, in particular, a front camera, a rear camera, and two side cameras of the vehicle. The front camera can, for example, be located in the area of ​​the front bumper of the vehicle and / or in the upper part of the windshield. The rear camera is located, for example, in the area of ​​the rear bumper and / or in the upper part of the rear window. The two side cameras can, for example, be located in the side mirrors of the vehicle. In a preferred example, all four surround-view cameras are used, so that the resulting bird's-eye view image describes a 360-degree view of the vehicle's surroundings, assuming that at least some of the vehicle's exterior walls are captured by the images from the four surround-view cameras. At least the two side cameras allow for the capture of the vehicle's exterior walls, including the doors and their surroundings. This illustrates how the method described above can be implemented using surround-view cameras already installed in the vehicle and bird's-eye view images typically generated by systems such as parking assistants. The use of a bird's-eye view image is particularly suitable when the vehicle has additional functions that involve generating this image. Such functions could include a distance warning system that displays potential obstacles around the vehicle. Furthermore, one embodiment provides that the method is automatically activated when at least one vehicle door is unlocked and / or opened. After activation, the method is repeated, at least temporarily, at predetermined time intervals. It can therefore be provided that as soon as the vehicle is unlocked, for example, using a mobile device, in particular a vehicle key and / or a smartphone, the method is already started, meaning the camera images are received and evaluated to determine the object information. Alternatively or additionally, the actual opening of at least one vehicle door may be required to activate, i.e., trigger, the method described above. The vehicle door can be opened manually or automatically. The system then repeatedly receives at least one camera image and determines the object information. For example, object information can be determined sequentially at predefined time intervals. These predefined time intervals can be, for example, 1 second, 2 seconds, 3 seconds, 5 seconds, 10 seconds, 15 seconds, or, in particular, 30 seconds. Time intervals between these values ​​are also possible. This approach is based at least on the understanding that once at least one vehicle door is unlocked and / or opened, it can be expected that the object will become trapped in the vehicle door, for example, because an occupant enters the vehicle and the object becomes trapped in the vehicle door when it is closed. In one example, it may be provided that after a predetermined time window of, for example, several minutes and / or after pressing the start-stop button without a previously detected jammed object, the procedure is terminated, i.e., deactivated. The process can therefore be selectively activated and / or terminated in appropriate situations where the object is likely to become jammed, in order to be energy-efficient and require little computational effort. Another aspect of the invention relates to a control device configured to carry out the method described above. It performs the method. The control device can be arranged in a motor vehicle, in particular, it can be encompassed by the motor vehicle. The control device includes, for example, a processor. This processor can include at least a microprocessor, microcontroller, FPGA (Field Programmable Gate Array), and / or DSP (Digital Signal Processor). Furthermore, it can contain program code, which can alternatively be referred to as a computer program product. The program code can be stored in a data memory of the processor. Another aspect of the invention relates to a motor vehicle configured to perform the method described above. The motor vehicle is, for example, a passenger car, a truck, a bus, a motorcycle, and / or a moped. The motor vehicle may include the described control device. Another aspect of the invention relates to a computer program product. The computer program product is a computer program. The computer program product comprises instructions that, when the program is executed by a computer, such as by the control devices, cause it to perform the corresponding steps of the method described above. The embodiments described in connection with the method according to the invention, both individually and in combination with one another, apply accordingly, where applicable, to the motor vehicle, the control device, and the computer program product according to the invention. The invention comprises combinations of the described embodiments. Figure 1 shows a schematic representation of a motor vehicle with an object trapped in a vehicle door, and Figure 2 shows a schematic signal flow graph of a method for checking whether at least one object is trapped in a closed vehicle door of a motor vehicle. In the figures, functionally identical elements are provided with the same reference symbols. Fig. 1 shows a motor vehicle 1 with several doors 2. The sketched motor vehicle 1 has four doors 2: two opposing doors 2 at the front of the vehicle 1 and two opposing doors 2 at the rear of the vehicle 1. The doors 2 are located on the outer walls 3 of the motor vehicle 1. In one example, an object 4, such as a piece of clothing belonging to an occupant of the motor vehicle 1, may be trapped in one of the doors 2. The vehicle 1 can have several surround-view cameras 5. Two side cameras in the side mirrors 6 of the vehicle 1, two front cameras in a front area 7, one of which is arranged in an upper area on the windshield 8, and a rear camera in a rear area 9 of the vehicle 1 are shown as examples. Other or additional surround-view cameras 5 are possible. For example, side cameras can be arranged additionally or alternatively on the outer walls 3 and not in the side mirrors 6. Using, for example, the side cameras, at least one of the front cameras, and the rear camera, a 360-degree view of the vehicle 1 can be captured. Each of the individual surround-view cameras 5 can capture at least a part of the vehicle 1's surroundings 10. The vehicle 1 can, in one example, have a control device 11. Fig. 2 shows a method for checking whether at least one object 4 is trapped in a closed vehicle door 2 of the motor vehicle 1. The method can be carried out using the control device 11. In a process step S1, at least one camera image 12, 13, 14, 15 is received. Here, for example, camera image 12 is captured by one of the front cameras, camera images 13, 14 by the side cameras, and camera image 15 by the rear camera and received in process step S1. The respective surround-view camera 5 transmits the respective captured camera image 12, 13, 14, 15 to the control device 11 so that it can carry out process steps S1 to S5 of the method. The capture of the camera images 12, 13, 14, 15 can also be understood as a process step of the method (not shown here). The individual camera images 12, 13, 14, 15 each describe at least a part of the surroundings 10 of the motor vehicle 1.They were detected by at least one of the surrounding cameras 5. In process step S2, a bird's-eye view image 16 of the surroundings 10 of the vehicle 1 is determined. If only parts of the entire surroundings 10 are described by the received camera images 12, 13, 14, 15, the bird's-eye view image 16 describes only these parts of the surroundings 10. The bird's-eye view image 16 describes at least one of the outer walls 3 of the vehicle 1, at least partially. It is therefore not intended that the bird's-eye view image 16 describes, for example, only the surroundings 10 of the vehicle 1 in the front area 7 and one outer wall of the vehicle 1 in the front area 7. The bird's-eye view image 16 is determined by applying a bird's-eye view image determination algorithm 17 to the at least one received camera image 12, 13, 14, 15. In a process step S3, at least one section 18 of the bird's-eye view image 16 can be determined in an example. At least one of the outer walls 3 of the vehicle 1 lies at least partially within this section 18. Here, for example, a section 18 is determined for each outer wall 3 of the vehicle 1. In a subsequent process step S4, only the determined sections 18 can then be considered. The section 18 of the bird's-eye view image 16 can be predefined, for example, by predefined image pixels or image pixel ranges. In process step S4, it is checked whether at least one object 4 is trapped in the closed vehicle door 2 of the motor vehicle 1 by applying an object recognition algorithm 20 to the determined bird's-eye view image 16. In one example, the object recognition algorithm 20 can only be applied to at least one section 18, or, in the example shown here, to the two determined sections 18 of the bird's-eye view image 16. If it is determined that at least one object 4 is trapped in the closed vehicle door 2 of the motor vehicle 1, the object information 19, which describes that at least one object 4 is trapped in the closed vehicle door 2 of the motor vehicle 1, is determined in process step S4. If no object 4 is trapped in the closed vehicle door 2 of the vehicle 1, object information 19 can also be determined, which then describes that no object 4 is trapped in the closed vehicle door 2 of the vehicle 1. Furthermore, the determined object information 19 can, in one example, describe at least the vehicle door 2 in which object 4 is trapped. It can therefore distinguish between the four different vehicle doors 2 in this case. In process step S5, for example, a warning message 21 can be issued in vehicle 1, indicating the trapped object 4, if object information 19 is determined that at least one object 4 is trapped in the closed vehicle door 2 of vehicle 1. Alternatively or additionally, in the same case, i.e., if object information 19 is determined that at least one object 4 is trapped in the closed vehicle door 2 of vehicle 1, driving of vehicle 1 can be prevented, at least temporarily; that is, a driving blockage 22 can be implemented. The object recognition algorithm 20 can comprise at least one neural network. This neural network can perform instance segmentation to recognize object 4 with pixel-level accuracy in the bird's-eye view image 16 or in at least one section 18. The neural network for instance segmentation can, for example, have a YOLACT architecture (YOLACT stands for "You Only Look At Coefficients"). In one example, as outlined here, four camera images 12, 13, 14, 15 can be received, which were captured by four different environmental cameras 5 of the vehicle 1. The procedure can always be automatically activated when at least one vehicle door 2 of the vehicle 1 is unlocked and / or opened. After activation, it can be repeated at predetermined time intervals, at least temporarily, for example, until the trapped object 4 is detected or until it is not detected and the vehicle 1 starts moving. Overall, the examples demonstrate a camera-based solution for detecting an abaya or other textiles that are caught in a vehicle door 2, using deep learning, i.e., a deep neural network. The method enables the detection and identification of articles of clothing (object 4) trapped in one of the closed vehicle doors 2. To detect trapped articles of clothing, the method uses the fisheye images from the surround-view camera system (camera images 12, 13, 14, 15) as input. In particular, the method functions perfectly with the existing camera mounting positions and orientations. No additional sensors are required for detecting the trapped articles of clothing. Furthermore, the warning can be displayed as a warning signal / haptic signal / alarm on an existing display device (notification message 21). A deep learning-based solution is proposed to detect and classify the presence of clothing items caught in the vehicle doors 2. The solution can indicate in which vehicle door 2 (left, right, front, or rear) the item of clothing is caught. The advantages of the proposed method include, at a minimum, ensuring the safety of the driver / occupants. The method addresses the following use cases: 1. A more realistic solution for autonomous driving to achieve Level 3 and higher autonomy. 2. Use of camera images 12, 13, 14, 15 and the existing sensor configuration (surrounding cameras 5). 3. An image-based solution, thus eliminating the need for an additional sensor. 4. An effective solution for detecting and classifying objects 4, particularly an abaya and / or other clothing, caught in the vehicle door 2. The procedure can be activated by default as soon as the vehicle 1 is opened, even remotely. It uses the cameras in the vehicle 1 (surround-view cameras 5) to search for trapped clothing (object 4). The procedure checks the status of all doors, whether they are open or closed. If they are closed and all surround-view cameras 5 are activated, the procedure generates a top-down view of the four camera images 12, 13, 14, 15 from the surround-view cameras (bird's-eye view image 16). In addition to the top-down view, mirror images to the left and right are also considered for processing; that is, camera images 13, 14 from the side cameras in the side mirrors 6, to ensure the accurate detection and identification of trapped clothing items of varying lengths and sizes. The four surround-view cameras can be different from the side cameras or they can include the side cameras.If two articles of clothing are detected trapped in one of the closed vehicle doors, a warning signal may be displayed, an alarm may be triggered, or a haptic signal may be sent to the driver or the relevant seats (notice message 21). In bird's-eye view image 16, the relevant area (section 18) is defined. The instance segmentation of trapped clothing is detected within the relevant area (section 18) defined in the images. For mirror images from the side cameras, the segmentation is performed on the corrected images after section 18 has been drawn. Section 18, defined in the top view and in the corrected mirror images, serves as input for the instance segmentation. A real-time instance segmentation architecture such as YOLACT is proposed and modified for the detection of clothing items caught in vehicle doors 2. The instance segmentation is capable of distinguishing different instances of the same object 4. It performs pixel-level classification to locate the individual objects 4 and improves the scope and detection capability of each individual object 4 in the image (bird's-eye view image 16). YOLACT is a widely used real-time instance segmentation framework that has been optimized for various applications. The trade-off between real-time implementation and detection accuracy is promising, which is why it is modified and used. YOLACT is a simple, fully convolutional model that can be trained on a single graphics processor and is used for real-time instance segmentation. It has two parallel tasks that can be assigned to form the final masks. The first task generates a set of prototype masks, and the second task predicts the coefficients of the instance masks. The final masks are generated by linearly combining the prototypes with the mask coefficients. The existing single-stage object recognition model is extended with an additional masking branch to generate instance segmentation of objects in real time. The first branch uses a fully convolutional network (FCN) to generate a set of image-sized prototype masks, independent of instances. The second branch predicts a vector of mask coefficients for each anchor by adding an extra header.Each instance that survives Non-Maximum Suppression (NMS) is created as a mask by a linear combination of the work of these two branches. The prototype generation branch (Protonet) is used to predict a set of k prototype masks for the entire image. Here, k refers to 5 classes. An FCN attached to a backbone feature layer, whose last layer has k channels, may be provided. The feature pyramid network (P3 layer) is used for the largest feature size and is the deepest. To improve performance with small objects, it is scaled down to 1 / nth dimensions of the input image, where n depends on the size of the smallest object to be detected. For example, in the Protonet architecture, four defined layers may be provided for the first branch of the YOLACT model. YOLACT can specify the feature size and channels for an image size of 550 x 550 pixels. The first two layers use 3 x 3 convolutional layers, while the third layer uses 1 x 1 convolutional layers.The image size is increased by upscaling, followed by a convolution operation. In addition to predicting the shape and class of objects 4, a third branch is added to predict k mask coefficients. Therefore, 4 + c + k coefficients are generated per anchor. The tanh function is used to produce more stable outputs. The output of the prototype branch and the mask coefficients are combined linearly, which can be implemented using sigmoid functions as shown below: where P is an hxwx k matrix of prototype masks and C is an nx k matrix of mask coefficients for n instances that survive NMS and thresholding. Loss functions that can be used when training the neural network are described below: The total loss used during model training comprises three losses: classification loss Lclc, bounding box regression loss Lbox, and mask loss Lmask. The bounding box regression loss function is proposed as DloU loss, which is a distance-based intersection-over-union loss for detecting small objects 4. The object 4 trapped in the vehicle door 2 results in a representation as a small object 4, so defining a loss function for detecting small objects 4 seems logical. It is defined as: where d is defined as a distance between the midpoints of a target bounding box from a dataset and a predicted bounding box, and C is defined as a distance between an upper-left corner of the predicted bounding box and a lower-right corner of the target bounding box from the dataset. The Kullback-Leibler divergence loss function can be used as a classification loss function, which is defined as follows: The mask loss function is defined as pixel-wise binary cross-entropy between composite masks M and the ground truth masks Mgt, defined as: Lmask = BCE(M, Mgt). The final loss function, defined as total loss, is the sum of all three losses: total loss = Lcls + Lbox + Lmask. To preserve small objects in the prototypes, the final masks are clipped during training using the bounding box of the basic truth; that is, Lmask is divided by the bounding box of the basic truth. During evaluation, the final masks are clipped using the predicted bounding boxes. QUOTES INCLUDED IN THE DESCRIPTION This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature US 2023 / 0339401 A1

[0003]

Claims

Method for checking whether at least one object (4) is trapped in a closed vehicle door (2) of a motor vehicle (1), for the motor vehicle (1), comprising: - Receiving at least one camera image (12, 13, 14, 15) describing at least part of an environment (10) of the motor vehicle (1) and captured by an environment camera (5) of the motor vehicle (1); - Determining a bird's-eye view image (16) of the environment (10) of the motor vehicle (1), wherein the bird's-eye view image (16) describes at least part of an outer wall (3) of the motor vehicle (1), by applying a bird's-eye view image determination algorithm (17) to the at least one received camera image (12, 13, 14, 15); - Checking whether the at least one object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1) by applying an object recognition algorithm (20) to the determined bird's-eye view image (16);and- if it is determined that at least one object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1), determine object information (19) that at least describes that at least one object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1). Method according to claim 1, wherein, if it is determined that no object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1), object information (19) is determined which describes that no object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1). Method according to one of the preceding claims, wherein, if the object information (19) is determined which at least describes that the at least one object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1), a warning message (21) is issued in the motor vehicle (1) indicating the trapped object (4). Method according to one of the preceding claims, wherein, if the object information (19) is determined which at least describes that the at least one object (4) is trapped in the closed vehicle door (2) of the motor vehicle (1), driving of the motor vehicle (1) is prevented at least temporarily. Method according to one of the preceding claims, wherein the object information (19) describes at least the vehicle door (2) in which the object (4) is trapped. Method according to one of the preceding claims, wherein after determining the bird's-eye view image (16) at least one section (18) of the bird's-eye view image (16) is determined in which at least one outer wall (3) of the motor vehicle (1) is at least partially located, and the object recognition algorithm (20) is applied only to the at least one determined section (18) of the bird's-eye view image (16). Method according to claim 6, wherein at least one section (18) of the bird's-eye view image (16) is fixed. Method according to one of the preceding claims, wherein the object recognition algorithm (20) comprises at least one neural network. Method according to claim 8, wherein the at least one neural network performs instance segmentation to recognize the object (4) pixel-accurately in the bird's-eye view image (16). Method according to claim 9, wherein the neural network for instance segmentation has a YOLACT architecture. Method according to one of the preceding claims, wherein four camera images (12, 13, 14, 15) are received which were captured by four different environmental cameras (5) of the motor vehicle (1), in particular by a front camera, a rear camera and two side cameras of the motor vehicle (1). Method according to one of the preceding claims, wherein the method is automatically activated when at least one vehicle door (2) of the motor vehicle (1) is unlocked and / or opened and is repeated at least temporarily at predetermined time intervals after activation. Control device (11) for a motor vehicle (1) which is designed to perform a method according to one of claims 1 to 12. Motor vehicle (1) which is designed to perform a method according to any one of claims 1 to 12. Computer program product comprising instructions which, when the program is executed by a computer, cause it to execute a method according to any one of claims 1 to 12.

Images