Method and system for assisting opening or closing of door
A computer-implemented method and system using image processing and logic-based algorithms for automated door operation in vehicles addresses the lack of efficient door automation, ensuring safe and adaptive door management by detecting environmental interference.
Patent Information
- Application Number
- JP2024068595
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-27
- Filing Date
- 2024-04-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-04-19
AI Technical Summary
Existing systems lack efficient automation for assisting in the opening and closing of vehicle doors, particularly in shared vehicles, where tasks such as managing door operations are often performed manually by drivers or conductors.
A computer-implemented method and system that utilizes image processing and logic-based algorithms to detect objects and determine interference indicators for door movement, using cameras to capture images of door operating areas and adjacent regions, and employing machine learning for robust object and person detection to ensure safe and automated door operation.
Enables reliable and adaptive automation of door opening and closing operations by considering environmental interference, enhancing safety and efficiency in vehicles by minimizing the risk of door obstruction.
Smart Images

Figure 2024159603000001_ABST
Abstract
Description
[Background technology]
[0001] 1. Field of the Invention The present disclosure relates to the field of automation, and in particular to a method and system for assisting the opening or closing of doors. The method and system may find application in all kinds of industries, including the automotive industry.
[0002] 2. Description of Related Technology In recent years, the demand for automation has increased across all fields of technology. In particular, the field of autonomous vehicles has experienced great developments to partially or fully automate the driving tasks that are usually performed by a driver. However, in vehicles, and more specifically in shared vehicles, such as buses, shuttles, trains, etc., the driver (or sometimes the conductor or pilot) usually performs tasks other than the pure driving task, such as managing the opening and closing of doors, deciding whether the vehicle can depart, assisting disabled people, dealing with emergency situations, and the like.
[0003] Therefore, there is a need for more automation, especially with regard to assisting in opening and closing the doors.
[0004] The following references disclose methods related to the field of background management in images. [Reference 1] Shanchuan Lin et al., Real-Time High-Resolution Background Matting, arXiv:2012.07810v1 Summary of the Invention
[0005] In this regard, the present disclosure provides a computer-implemented method for assisting in opening or closing a door, the method comprising: - acquiring at least one image showing at least one region of interest, the at least one region of interest comprising a door movement area or an area adjacent to the door movement area; - detecting in at least one image whether at least one object is present in a region of interest; determining at least one interference indicator for opening or closing the door based on the detection; The present invention relates to a computer-implemented method,
[0006] The steps of the method may be performed by one or more processors. The method is hereinafter referred to as an assisted method.
[0007] Acquiring at least one image may include, for example, acquiring an image through an image acquisition module, such as a camera, or obtaining previously acquired images from a database, e.g., a local server, a remote server, or the like. Hereinafter, unless otherwise stated, "image" refers to at least one image. More generally, as used herein, for brevity, unless the context indicates otherwise, "a," "an," and "the" refer to "at least one" or "each," and are intended to include the plural form as well. Conversely, the general use of the plural includes the singular form as well.
[0008] In some embodiments, the at least one image comprises a plurality of consecutive frames of a video clip. Alternatively, the at least one image may comprise a plurality of non-consecutive frames of a video clip, for example, selected at a given sampling frequency (e.g., every third frame of the video clip). Still alternatively, the at least one image may comprise one or more static images, for example, photographs. Using frames from a video clip allows the detection information to be verified over time, thereby providing robustness to the method.
[0009] Each acquired image may show one or more regions of interest. A region of interest (ROI) is an area of the door environment in which an element of interest for determining interference indicators can be seen. The region of interest comprises a door movement area or an area adjacent to a door movement area. In this regard, a door movement area is an area that intersects with a door during opening or closing of said door. The door movement area does not necessarily cover the entire travel distance of the door.
[0010] In this disclosure, object should be understood as a general term meaning any obstacle to a door, including a human being, an animal, a plant or an object.
[0011] The interference indicator describes how the door's environment, particularly objects within the region of interest, may interfere with the opening and / or closing of the door (hereafter shortened to open / close).
[0012] The method thus allows to determine, based only on the image, to what extent objects around the door interfere with a possible door movement (e.g. opening / closing) and to output an interference indicator based on which additional actions may be taken, such as triggering the opening / closing of the door. The proposed method thus provides reliable assistance for the opening / closing actions based on simple inputs. Thus, the task of opening / closing the door can be handled automatically.
[0013] Optionally, the method is performed in real time to assist in opening / closing the door based on current conditions regarding the environment around the door.
[0014] Optionally, the determination includes taking the multiple detections as inputs and applying a logic-based algorithm. The logic-based algorithm is, for example, a rule-based or probability-based algorithm that applies the rules or probabilities to the multiple inputs, i.e., the multiple detections, to determine an output. The multiple detections may include detections of each of the multiple objects in at least one image, or detections in multiple images (e.g., one detection per image). By combining the detection method and the logic-based algorithm based on the multiple detections, the method is not only more reliable because it is based on multiple detections, but also simpler because the detections are combined by the logic-based algorithm. This avoids the need for complex models to handle a wide variety of situations. In contrast, combining the detections through the logic-based algorithm allows to leverage the advantages of simple detections and the robustness of logic-based reasoning that brings human insight to the method. The architecture also brings self-adaptability to the assistance method.
[0015] Optionally, the at least one image includes a first image showing a first side of a door threshold and a second image showing a second side of the door threshold opposite the first side. The door threshold is defined by the location of the door when it is closed. The threshold may be embodied by a frame, a sill, or any other means, depending on the door type. By providing these, both sides of the door threshold are taken into account for the determination of the interference indicator.
[0016] Alternatively or additionally, a single image may show both sides of the door sill.
[0017] Optionally, the image is an RGB (red / green / blue) image, or more generally, a single-channel or multi-channel image. An RGB image of a scene comprises pixels having red, green and blue values to form colors (including black and white or grayscale) that represent the colors that the scene has. A single-channel image is an image in which each pixel has a color or intensity represented by a single value, and a multi-channel image is an image in which each pixel has a color or intensity represented by multiple values. Grayscale is an example of a single-channel scale, while RGB is an example of a multi-channel scale.
[0018] Optionally, the images are acquired by a fish-eye camera, which is known in the art and allows to acquire a large field of view.
[0019] Optionally, the at least one interference indicator comprises a first interference indicator determined based on the first image and a second interference indicator determined based on the second image, and the method further comprises selecting one of the first interference indicator and the second interference indicator corresponding to the highest interference level. The interference level is represented as high if the detected object obstructs or impedes the opening / closing of the door, while the interference level is represented as low if the detected object, if any, has little or no interaction with the door during the opening / closing movement. For example, the first interference indicator and the second interference indicator may be represented on the same scale, larger if the interference level is high, and the method may select the highest of the first interference indicator and the second interference indicator.
[0020] The first interference indicator and the second interference indicator may enable the method to take into account multiple object configurations and determine an interference indicator based on the highest detection risk for interference between the door and its environment.
[0021] In an embodiment, the final decision to enable or trigger the opening / closing of the door may be based on the selected interference indicator.
[0022] Optionally, the first image at least partially depicts a second side of the door sill, or the second image at least partially depicts a first side of the door sill, in which embodiment at least one of the images depicts both sides of the door sill, which increases the reliability of the method.
[0023] Optionally, the detection further includes detecting whether a person is present in the region of interest. That is, detecting whether a person is present in the region of interest is performed separately from detecting whether an object is present in the region of interest. As a result, if a person is present in the region of interest, the person is expected to be detected by both object detection and person detection. Person detection allows identifying a person as a person with possible specific implications or attention, while object detection may be more general and not classify the object. In addition, having both object detection and person detection introduces redundancy in the method, but this also leads to increased robustness.
[0024] Optionally, people are detected by skeleton detection, which is a simple and efficient method for detecting humans.
[0025] Optionally, the detection includes extracting foreground elements in at least one image against a reference background. This may be applied to object detection and / or person detection. The reference background comprises elements of the door environment (e.g., the rest of the vehicle, roads, trees, buildings, etc.) that may or may not be static with respect to the door or door threshold, and serves as a reference for the detection of foreground elements. Foreground elements are elements added on top of the background, e.g., occupants, other vehicles, objects on the ground, etc., that may interfere with the opening / closing of the door.
[0026] By extracting foreground elements relative to a reference background, the method can adapt to different backgrounds (e.g., different locations, different lighting, etc.) and consider only relevant elements of the image. Image-based assistance is therefore more reliable.
[0027] Optionally, the extraction is performed by a machine learning model. The machine learning model may comprise one or more mathematical expressions configured to process the image to output information representative of the foreground elements. The weights of the machine learning model may be iteratively updated by a training method. The machine learning model may comprise an artificial neural network.
[0028] Optionally, the extraction includes computing a mask of foreground elements in at least one image, which indicates the foreground elements while masking other pixels of the image. The mask computation itself is known in the field of image processing and is simple to use. As a result, the proposed method is efficient and robust.
[0029] Optionally, the method further comprises updating the reference background based on at least one image if no person is detected in the at least one image. If no person is detected in an image, it is assumed that the image represents the actual background except for possible objects that are easier to detect and less likely to move by themselves. Thus, the absence of people is exploited to update the background. Updating the background makes it possible to take into account any changes in the environment and ultimately improve the results of the method.
[0030] Optionally, the method includes initializing the reference background. The initialization may be performed if the reference background has not been set before or if the reference background has not been set for a certain period of time. The initialization may be performed automatically.
[0031] The present disclosure further provides a system for assisting in opening or closing a door, the system comprising: an acquisition module for acquiring at least one image showing at least one area of interest, the at least one area of interest comprising a door movement area or an area adjacent to the door movement area; a detector for detecting in at least one image whether at least one object is present in a region of interest; a determination unit for determining, based on the detection, at least one interference indicator for opening or closing the door; The present invention relates to a system comprising:
[0032] A system, hereinafter referred to as assistance system, may be configured to carry out the assistance method described above and may have some or all of the features described above. The assistance system may have a computer hardware structure.
[0033] The present disclosure is further directed to a vehicle comprising at least one door and the above-mentioned system for assisting in opening or closing the door. The vehicle may be a motor vehicle, such as a car, a bus, a shuttle, or other vehicle, such as a train, an airplane, etc.
[0034] Optionally, the vehicle is an autonomous vehicle. The assistance system may provide non-driving functions of the autonomous vehicle.
[0035] The present disclosure further relates to a computer program set comprising instructions for carrying out the steps of the above-mentioned support method when said program is executed by at least one computer, said set of programs may be in source code, object code, or intermediate code between source code and object code, for example in a partially compiled form, using any programming language, or in any other desired form.
[0036] The present disclosure further relates to at least one computer readable recording medium having recorded thereon at least one computer program comprising instructions for performing the steps of the above-mentioned assistance method. The recording medium may be any entity or device capable of storing a program. For example, the medium may include a storage means such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or a magnetic storage means, for example a diskette (floppy disk) or a hard disk.
[0037] Alternatively, the recording medium may be an integrated circuit in which the program is embedded, the circuit being configured to perform, or for use in the performance of, the method. [Brief description of the drawings]
[0038] The features, advantages, and technical and industrial importance of preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements, and in which: [Figure 1] FIG. 1 is a flow chart illustrating the calibration portion of an assistance method according to an embodiment. [Diagram 2] FIG. 2 is a flowchart showing the determination part of the support method according to the embodiment. [Diagram 3] FIG. 3 is a diagram of a vehicle according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0039] The method for assisting in opening or closing a door according to an embodiment will be described with reference to Figures 1 and 2. In this embodiment, the door is a vehicle door. However, the following description can be transposed to other types of doors.
[0040] As shown in Fig. 1, the assistance method includes obtaining at least one image showing an area of interest, the at least one area of interest comprising a door operation area or an area adjacent to the door operation area. In particular, Fig. 1 shows two such images, an inner image 10 and an outer image 11. The inner image 10 (first image) may show a first side (inner) of a door sill, e.g., a vehicle cabin, and the outer image 11 (second image) may show a second side (outer) of the door sill, which is opposite to the first side.
[0041] In particular, the inside image 10 shows a region of interest inside the vehicle, e.g., an area of the vehicle ground where a person walks before exiting through a door. In another example, the inside image 10 shows an area where a door moves, e.g., for a bus, the door may open towards the inside of the vehicle. Such an area is referred to as an inside ROI (region of interest).
[0042] Conversely, the exterior image 11 shows a region of interest outside the vehicle, e.g., an area of the exterior ground where a person walks through a door before entering the vehicle. In another example, the exterior image 11 shows an area where a door moves, e.g., for a vehicle such as a car, the door generally opens toward the outside of the vehicle. Such an area is referred to as an exterior ROI (Region of Interest).
[0043] Although the inside image 10 and / or the outside image 11 (hereinafter referred to as "images 10, 11") are described herein as single images, they may each comprise multiple images, e.g., videos, rather than a static image (e.g., a photograph). In addition, the present disclosure is not limited to a single inside image and a single outside image, and more or fewer images may be envisioned. Furthermore, rather than having images from two or more viewpoints (here, one camera inside the vehicle and one camera outside the vehicle), the assistance method may use only one camera, which preferably shows at least the inside of the vehicle, or both the inside and the outside through a window or an open door.
[0044] The images 10, 11 may be acquired by a camera, for example a fish-eye camera, which may provide distorted images, but the images 10, 11 may not require any correction before being processed by the aiding method.
[0045] As indicated above, at least one of the inside and outside images may show both sides of the door sill. In this example, it is assumed that the inside image (first image) 10 at least partially shows the second side (outside) of the door sill. In such a situation, the region of interest may overlap the door sill. In the following, the region of interest that is outside the vehicle but visible from the inside is referred to as the inside-outside ROI.
[0046] In step 12, the assisted method acquires an inner image 10. In some cases, part or all of the video may be acquired in step 12. In step 14, the assisted method determines whether the ROI configuration has already been calibrated. If the ROI configuration has already been calibrated (YES), the assisted method proceeds to step 32 of loading an inner ROI based on the inner image 10 and step 34 of loading an inside-outer ROI. Steps 32, 34 may include cropping the inner image 10. The ROI configuration may be loaded from memory.
[0047] However, if the ROI configuration is not calibrated (NO at step 14), the assisted method enters into an ROI calibration process. At step 16, the assisted method reads the inner image 10 for calibration. In some cases, for example, if the inner image 10 comprises multiple images (a video or other sequence), step 16 may include reading a frame thereof. The assisted method then proceeds to step 18.
[0048] Step 18 includes optional cropping to limit the total computational requirements. For example, step 18 includes cropping the image 10 to the available region of interest. In this example, there are two regions of interest in the inner image 10, namely the inner ROI and the outer ROI, and thus two cropped versions of the image 10. Cropping the image 10 to maintain the region of interest can be done manually or automatically, for example based on the detection of feature points in the image. For example, the cropped region of the image 10 can have a simple shape, such as a polygonal, circular, or elliptical shape.
[0049] Possibly, after cropping, a configuration of each ROI may be created. Step 20 involves creating an inner ROI configuration, while step 22 involves creating an inside-outer ROI configuration. The ROI configuration may comprise the position, size and / or shape of the respective ROI in the (possibly cropped) inner image 10. In contrast to the cropped image, the shape of the ROI is not a simple shape, but may be any possible shape. These steps 20, 22 result in an inner ROI 24 and an inside-outer ROI 26, respectively.
[0050] The inner ROI 24 and inner-outer ROI 26 may be saved in respective save steps 28, 30. Once the ROI configurations are saved, the ROI calibration process proceeds to the load steps 32, 34 already described.
[0051] The exterior image 11 may be processed in the same way, so that the full calibration is not described again here. For example, steps 13, 15, 17, 19, 21, 29, 33 of acquiring the exterior ROI 25 may correspond to the above-mentioned steps 12, 14, 16, 18, 20, 28, 32 of acquiring the interior ROI 24. Of course, steps corresponding to the interior-exterior ROI may not be performed on the exterior image 11, which does not show the interior of the vehicle. However, more generally, multiple regions of interest may be loaded and, if necessary, calibrated in each acquired image.
[0052] The door itself may be visible in at least one of the ROIs.
[0053] To prepare for object detection, as described below, the assisted method may determine in step 36 whether a reference background has been calibrated, for example based on the inner image 10. Step 36 may follow step 12 described above. If the reference background has already been calibrated (YES), the assisted method proceeds to step 48, which loads the reference background. The reference background may be loaded from memory.
[0054] However, if the reference background is not calibrated (NO in step 36), the aiding method enters into a reference background initialization process. First, in step 38, the aiding method reads the interior image 10 for calibration. In some cases, for example if the interior image 10 comprises multiple images (video or other sequence), step 38 may involve reading a frame thereof. At this stage, it may be verified whether the vehicle is empty, for example whether no person is present inside the vehicle. Indeed, if a person was present in the vehicle, the interior image 10 would not represent only background data.
[0055] In step 12, possibly a video or multiple images may be loaded. Thus, if a person is detected in one of the frames, step 38 may read another frame of the same video so that the reference background can be properly initialized. If no person is detected in the current frame, the aided method proceeds to step 40. At least one of the frame analysis techniques such as motion estimation and object detection may be employed to identify and extract the most representative empty cabin frame from the multiple available frames. Several frames may be taken into account to create a reference background model as described below.
[0056] Step 40 involves cropping a portion of the inner image 10 that represents a fixed background. Portions that do not represent a fixed background may be excluded to reduce computational requirements. In this example, the reference background may include any of the vehicle ground, vehicle seats, vehicle side walls, etc., and conversely, the cropped region may exclude the vehicle windows, since the scenery recognizable through the windows may vary. Thus, step 40 involves cropping the reference background.
[0057] After cropping, a reference background model may be created that corresponds to the cropped reference background. Step 42 involves creating a reference background model. Any background modeling technique may be used, such as the background modeling technique of [Reference 1]. This step 42 results in a reference background model 44. For example, creating the reference background model from multiple frames may include at least one of aligning the frames, correcting for any misalignment or disparity, and averaging pixel values to reduce noise to capture a more accurate representation of the cabin background.
[0058] The reference background model 44 may be saved in a save step 46. Once the reference background model 44 has been saved, the reference background initialization process proceeds to a load step 48, previously described.
[0059] Although specific examples are described, the ROI calibration process and the reference background initialization process may be varied from the examples, for example, some steps may be omitted (e.g., cropped), other steps may be added, or the described functions may be performed in a different manner.
[0060] Figure 2 shows steps of the support method which follow the steps described with reference to Figure 1. That is, step 50 in Figure 2 follows steps 32, 34 and 48, while step 51 follows step 31, as indicated by the dashed arrows in Figures 1 and 2.
[0061] FIG. 2 shows an example of how the assistance method detects whether at least one object is present in a region of interest in an acquired image.
[0062] Specifically, in step 50, the inner image 10 is acquired. In the case of a video, step 50 may include reading a particular frame of the video. If the assistance method is applied in real time, step 50 may read the most recent frame of the inner image 10. The method then proceeds to step 52.
[0063] In step 52, the content of the inner ROI in the inner image 10 is extracted based on the inner image 10 (or frame) acquired in step 50 and the inner ROI configuration loaded in step 32. The assisted method then proceeds to step .
[0064] Step 54 involves extracting foreground elements in at least one image (here the current frame) relative to the reference background loaded in step 48. This extraction may be performed by a machine learning model, for example according to [Reference 1]. More generally, the extraction may involve computing a mask of the foreground elements in the image.
[0065] The assisted method then proceeds to step 56, where whether an object is present in the inner ROI is determined based on the extracted foreground elements. For example, step 56 may include applying the mask of step 54 to the inner image 10 (or a given frame thereof) taking into account the inner ROI configuration of step 32 to determine whether a foreground element (object) is present in the inner ROI.
[0066] Furthermore, the assistance method further comprises detecting whether a person is present in the region of interest, here an area of interest outside the vehicle. For the interior image, person detection may be performed from the image acquired in step 50. In particular, from step 50, the assistance method proceeds to step 58, possibly in parallel with step 52.
[0067] In step 58, the contents of the inside-outer ROI in the inside image 10 are extracted based on the inside image 10 (or frame) acquired in step 50 and the inside-outer ROI configuration loaded in step 34. The assisted method then proceeds to step 60.
[0068] At step 60, person detection itself is performed to determine whether a person is present in the inner ROI and the inner-outer ROI. For example, a person may be detected by skeleton detection, e.g., 2D skeleton detection, which is known in the art of computer vision.
[0069] The assisted method further proceeds to step 56. In view of the above, step 56 collects general object detection and person detection results to determine whether an object or person is present in the inner ROI.
[0070] From step 60 above, the assistance method also proceeds to step 62, where it is determined whether a person is present in the Inside-Outside ROI based on the person detection of step 60 and the Inside-Outside ROI configuration of step 34. This determination may be expressed as a first interference indicator, e.g., a numerical value representing the interference level between the detected person and the Inside-Outside ROI.
[0071] Optionally, step 60 also leads to step 64, where it is determined whether no people are present in the vehicle (e.g., in the entire inner image, or at least in the inner ROI). If the vehicle is indeed empty (YES), the method proceeds to step 66, where the reference background is updated based on the image. For example, as an update, the supporting method may use steps 40-46 described with reference to FIG. 1, using the image obtained from step 50 instead of the image obtained from step 38. However, other methods are possible. For example, the update may utilize a change detection algorithm and / or scene analysis to identify and introduce a new empty cabin frame to the existing background model.
[0072] Updating the reference background when the vehicle is found to be empty (e.g., no people inside) allows for taking into account changes in the vehicle's environment, such as light variations (e.g., day / night, sunny / cloudy weather, etc.) or changes in accessories on the vehicle, allowing the method to maintain high accuracy when environmental conditions change.
[0073] The outer image 11 may be processed in the same way as the inner image 10, so that this processing is not described again here. For example, steps 51, 59, 61, and 63 of detecting objects, particularly people, in the outer ROI may correspond to the above-mentioned steps 50, 58, 60, 62 of detecting people in the inner-outer ROI. In step 63, the determination of whether a person is present in the outer ROI may be represented as a second interference indicator.
[0074] The assistance method includes determining, based on the detection, at least one interference indicator for opening or closing the door, i.e. the determinations of steps 56, 62 and 63 are taken into account to derive whether the door can be opened or closed.
[0075] In this embodiment, the multiple detections from steps 56, 62, 63 for the inner ROI, inner-outer ROI, and outer ROI, respectively, are provided as inputs to a logic-based algorithm. The logic-based algorithm may comprise one or more rules that combine the results of the detection steps 56, 62, 63.
[0076] In this embodiment, for example, step 56 is followed by step 64, which determines whether an object is detected in the inner ROI. Further, in step 66, it is determined whether a person is detected in the inner ROI. In the example, if either a person or an object is detected in the inner ROI in step 64 or 66 (YES), the assistance method further proceeds to step 68, which issues an interference indicator indicating that the inner ROI is occupied. Thus, something is considered to interfere with the possible door movement, and therefore the door may be prevented from opening or closing.
[0077] Conversely, if neither an object is detected in step 64 nor a person is detected in step 66 (NO), the assistance method proceeds to step 70 and issues an interference indicator indicating that the inner ROI is vacant. Thus, the door may be opened or closed without interference with any person or object inside the vehicle.
[0078] With respect to the outside of the vehicle, the first indicator of step 62 and the second indicator of step 63 are combined to determine whether a person is detected in the inside-outside ROI or the outside ROI in step 72. If a person is detected (YES), the assistance method further proceeds to step 74 and issues an interference indicator indicating that the ROI outside the vehicle is occupied. Thus, something is deemed to be interfering with a possible door movement and therefore the door opening or closing may be prevented.
[0079] Conversely, if no person is detected in the inside-outside ROI or the outside ROI (NO), the assistance method proceeds to step 76 and issues an interference indicator indicating that the ROI outside the vehicle is vacant, so the door may be opened or closed without interference with any person or object outside the vehicle.
[0080] This illustrates how the assistance method may select as an interference indicator one of the first and second interference indicators that corresponds to the highest interference level, for example using a Boolean OR.
[0081] Conflicting decisions in detection can be handled in different ways. In an example, the highest interference level can be considered so as not to take any risk regarding door movement, as described above. In another example, if the outer ROI does not show human detection but the inside-outer ROI shows human detection, it can be decided to ignore the current frame and read another frame for confirmation.
[0082] In the above assistance method, it is described that object detection is performed inside the vehicle, while person detection is performed both inside and outside the vehicle. Although other embodiments are envisioned, this embodiment is advantageous to the extent that the environment inside the vehicle generally does not change much, which makes object detection easier, but the environment outside the vehicle generally changes a lot, such that object detection may not be reliable. In contrast, person detection works well either inside or outside the vehicle.
[0083] The interference indicators of steps 68 or 70 on the one hand, and 74 or 76 on the other, may be recombined to determine whether the door is open or closed from both an inside perspective and an outside perspective. Any combination of such interference indicators, such as those described above, is contemplated.
[0084] After issuing the interference indicators at steps 68, 70, 74, 76, the aiding method may loop back to steps 50, 51 to read the next frame of the respective inner and outer videos. In doing so, for example, the aiding method may be implemented in real time.
[0085] 3 shows a vehicle 80 comprising a door 82 and an assistance system 84 for assisting in opening or closing the door 82. The system 84 comprises an acquisition module 86 for acquiring at least one image showing at least one region of interest, the at least one region of interest comprising a door operation area or an area adjacent to the door operation area, a detector 88 for detecting in the at least one image whether at least one object is present in the region of interest, and a determination unit 90 for determining at least one interference indicator for opening or closing the door 82 based on the detection.
[0086] The assistance system 84 may include electronic circuits, processors (shared, dedicated, or group), combinatorial logic circuits, memory for executing one or more software programs, and / or other suitable components for providing the described functionality. In other words, the assistance system 84 may be a computing device. The assistance system 84 may be connected to a memory (recording medium), which may store data, such as a computer program, that, when executed, performs the assistance method according to the present disclosure.
[0087] The assistance system 84, in particular the acquisition module 86, has an input for receiving a digital image or a stream of digital images. In particular, the assistance system 84, in particular the acquisition module 86, may be connected to an optical sensor, in particular one or more digital cameras 92, for example the above-mentioned fish-eye cameras. The digital cameras 92 are arranged in such a way that they can record a scene, in particular output digital data to the assistance system 84, in particular the acquisition module 86.
[0088] If the method is performed in real time, the at least one image may be acquired in real time by a video acquisition module, e.g. an image acquisition module such as a camera or the like (video camera, photo camera, etc.). Alternatively or additionally, e.g. in the case of post-processing of a film process, the at least one image may be acquired in advance and later acquired by the assistance system 84.
[0089] Although the present disclosure refers to certain preferred embodiments, modifications to the examples may be provided without departing from the overall scope of the invention as defined by the claims. In particular, although the individual steps of the assistance method are described in a particular manner, individual steps or subgroups of steps can be modified to provide the same functionality. Furthermore, although described in the context of an autonomous vehicle, the assistance method and system may be implemented for conventional vehicles, for example, to prevent a vehicle occupant from opening a door from the inside when another vehicle (car, bicycle, etc.) is overtaking. More generally, individual features of the various embodiments shown / described may be combined in additional embodiments. The description and drawings should therefore be regarded as illustrative, and not restrictive.
Claims
1. 1. A computer-implemented method for assisting in opening or closing a door, comprising: - acquiring at least one image showing at least one area of interest, said at least one area of interest comprising a door movement area or an area adjacent said door movement area; - detecting in said at least one image whether at least one object is present in said region of interest; - determining, based on said detection, at least one interference indicator for opening or closing said door; 4. A computer-implemented method comprising:
2. The method of claim 1 , wherein the determining comprises applying a logic-based algorithm taking a plurality of the detections as input.
3. 3. The method of claim 1 or 2, wherein the at least one image includes a first image showing a first side of the door threshold and a second image showing a second side of the door threshold opposite the first side.
4. 4. The method of claim 3, wherein the at least one interference indicator comprises a first interference indicator determined based on the first image and a second interference indicator determined based on the second image, the method further comprising selecting one of the first interference indicator and the second interference indicator that corresponds to a highest interference level.
5. 4. The method of claim 3, wherein the first image at least partially depicts the second side of the threshold of the door or the second image at least partially depicts the first side of the threshold of the door.
6. The method of claim 1 or 2, wherein the detection further comprises detecting whether a person is present in the region of interest, optionally the person being detected by skeleton detection.
7. The method of claim 1 or 2, wherein said detection comprises extracting foreground elements in said at least one image against a reference background, optionally by means of a machine learning model.
8. The method of claim 7 , wherein the extracting comprises computing a mask of the foreground elements in the at least one image.
9. The method of claim 7 , further comprising: if no person is detected in the at least one image, updating the reference background based on the at least one image.
10. The method of claim 7 , further comprising initializing the reference background.
11. A system for assisting in the opening or closing of a door, comprising: an acquisition module for acquiring at least one image showing at least one area of interest, said at least one area of interest comprising a door movement area or an area adjacent to said door movement area; a detector for detecting in said at least one image whether at least one object is present in said region of interest; a determination unit for determining, based on said detection, at least one interference indicator for opening or closing said door; A system comprising:
12. A vehicle comprising at least one door and a system according to claim 11 for assisting in opening or closing said door, optionally said vehicle being an autonomous vehicle.
13. A computer program set comprising instructions for performing the steps of the method according to claim 1 or 2 when said program set is executed by at least one computer.
14. 3. At least one computer readable storage medium having recorded thereon at least one computer program comprising instructions for performing the steps of the method according to claim 1 or 2.
Citation Information
Patent Citations
Device for controlling opening and closing operation of vehicle door
JP2007153193A
Door opening / closing device for vehicle
JP2014080139A
Opening and closing system
JP2018009386A
Control device of open / close body on vehicle and method
JP2019090246A
Image processing device and image processing method
JP2019133446A
